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Abstract 


The Educational Quality Indicators (EQI) initiative was a three-year collaboration 
between Alberta Education and twelve school jurisdictions to develop indicator systems 
to measure the success of the educational enterprise. Its goal was to assist the provincial 
government and school jurisdictions in providing better information to improve planning, 
policy and decision making. Towards this end, ten collaborative action research projects 
were initiated in 1989 and concluded in 1992. These projects developed indicator 
systems that include a broad range of outcomes, methods to collect, analyze and interpret 
data, and ways to report and communicate the information to different audiences. 


Each project focused on specific aspects of educational improvement and accountability. 
The projects can be grouped into three major areas — system accountability, teaching and 
learning, and alternative student assessment strategies. 


Three projects focused on accountability. The Lacombe/Rocky Mountain project 
developed a comprehensive school system review process. Grande Prairie developed 
school and district profiles, and Lethbridge Public developed a collaborative model for 
school and program evaluation. 


Two projects examined teaching and learning. Spirit River identified 26 behaviors 
of effective teaching and prepared a teacher performance baseline for the division. 
Fort McMurray Catholic implemented the principles of outcome-based education 
in mathematics. 


Five projects examined alternative ways to measure student success. Edmonton Public 
developed mechanisms for portfolio assessment and student growth. Calgary Public 

and Catholic developed indicators of quality student performance in art and mathematics. 
Brooks explored ways to measure responsible student behavior. Fort McMurray Public 
developed a three-level approach to measuring desirable student social outcomes. 
Lethbridge Catholic developed a taxonomy of student affective behaviors. 


Achieving Quality is the final report of the Educational Quality Indicators initiative. 
It includes a summary of the initiative and each of the ten projects. A final chapter 
summarizes the major accomplishments, proposes potential provincial indicators, 
and suggests further steps in expanding the use of indicators in school jurisdictions 
in Alberta. 
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The Educational Quality Indicators Initiative 


Alberta Education 


The Educational Quality Indicators (EQI) initiative was a collaboration between 
Alberta Education and twelve school jurisdictions to develop and implement 
indicator systems to measure the success of the educational enterprise. Its goal 
was to assist the provincial government and school jurisdictions in providing 
better information to improve planning, policy and decision making. Towards 
this end, ten concurrent collaborative action research projects were initiated in 
1989 and concluded in 1992. These projects developed indicator systems that 
include a broad range of outcomes, methods to collect, analyze and interpret 
data, and ways to report and communicate the information to different audiences. 
Achieving Quality is the final report of the EQI initiative. 


Introduction 


Background 


A new wave of educational reform began in the 1980s. Perceived shortcomings 

in education, and international competition in ell sectors, served as a catalyst for 
the current accountability and reform movements in Canada and the United States, 
and indeed around the world. These reforms resulted in many organizations 
developing or implementing what have become known as educational indicator 
systems. Indicators can describe education in a way that permits meaningful 
interpretation to take place, leading to educational improvement and accountability. 


A major educational reform effort in Alberta was stimulated by a series of 
changes in the political, economic and social climate of the province. In 1982, 
Alberta Education introduced the Achievement Testing Program at grades 3, 6 
and 9, and in 1984, reinstated the Diploma Examination Program at grade 12 as 
part of the requirements for high school graduation. Annual provincial reports 
provide information about test results at the provincial level. 


In 1984, the Government of Alberta introduced a number of other initiatives 

to improve education in the province. These included the School Act Review, 
the Management and Finance Plan, the Review of Secondary School Programs, 
and the introduction of five evaluation policies (student, teacher, program, school 
and school system). As a result, the present management approach attempts to 
balance the inputs, processes and outcomes of the educational enterprise. 
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Two primary beliefs influenced the reform initiatives in Alberta. First, the student 
is the central reason for schooling, and second, education is a purposeful endeavor 
that can be assessed. To assist in ensuring that these beliefs became operational, 
Alberta Education promoted a results-based approach to education to provide a 
better balance among inputs, processes and results. 


The EQI initiative was one of the projects designed to promote results-based 
education in Alberta. A significant feature was that it introduced a collaborative 
model to the provincial reform initiatives. EQI participants became members of 
a provincial committee which discussed and recommended directions to Alberta 
Education in achieving results-based education. The province will synthesize the 
results and, together with its other initiatives, work toward a provincial indicator 
system. 


EQI was designed to produce a set of indicators, appropriate standards and 
accompanying methods to measure the performance of the educational system 

in the province. Indicators provide information to assist in assessing the quality of 
educational programs and the delivery system by focusing on student outcomes. 
The proposed system of measuring success has taken into consideration and 
reflects government policy and the goals of schooling. It addresses two essential 
questions: 


1. Are students learning to their potential? 


2. Is the educational system supporting student learning effectively and 
efficiently? 


Rationale 


The current educational reform movement led to expectations for greater 
accountability and increased monitoring and evaluation of schools and systems. 
Indicator systems help to assess the effectiveness and efficiency of the educational 
enterprise, to improve education, and to provide a mechanism for accountability. 
The expected results over the three years of the EQI initiative were: 


¢ indicator systems to include a broad range of student outcomes, points of 
reference for comparing results, and an interpretative framework to describe 
variation among students and schools; 


¢ methods to collect, analyze and interpret the information provided by the 
indicators; and 


* ways to report and communicate the information to different audiences. 


A fundamental principle of the EQI initiative was that no single indicator, or 
even group of indicators, could fully describe the complexity of education. 
Any system would include many indicators, measured by both quantitative 
and qualitative methods, for selected dimensions. An indicator system is 
intended to enhance information about education for improved action in 
planning, policy and decision making. 
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The use of multiple indicators repeated over time can describe the complexity 

of education and provide a good picture of performance. Information available 
through documentation, student assessment, surveys and observation, can 
describe the effectiveness and efficiency of the operation. As there are multiple 
users of information, it is also necessary to determine what type of information the 
different audiences require and to tailor information reports to their specific needs 
(Bock & Mislevy, 1988). 


Conceptual Framework 


The conceptual framework for the Educational Quality Indicators initiative guided 
the direction of the ten collaborative action research projects and the discussion of 
results-based education within Alberta Education. Educational indicators provide 
information about a system's current functioning, suggest whether progress is 
being made and warn of any potential problems (Oakes, 1986). While indicators 
cannot describe a system completely, they do provide a picture of existing 
conditions which can inform planning, policy and decision making. 


An educational indicator system should provice: 
¢ information that is feasible to gather, valid and useful for policy decisions; 
¢ logical clusters of indicators to increase the system's interpretative power; 


¢ points of reference such as a previous value, the value of a comparison group 
or some socially determined standard; 


* measures of the ubiquitous features of schooling that are: enduring, easily 
understood, feasibly measured and generally accepted as valid and reliable 
statistics; 


* areporting system that is accurate and timely (Oakes, 1986; Selden, 1988). 


A four-dimensional model of education was developed to guide the direction of 
this initiative. It consists of partners (schooling, family and society), conditions 
(context, inputs and processes), student outcomes (cognitive, affective and 
behavioral) and time (grades 3, 6, 9 and 12) (McEwen & Zatko, 1989). The 
model draws on the work of Carroll (1963), Hymel (1988), Walberg (1984), 
Oakes (1986) and Shavelson et al (1987). The EQI model extends the work of 
the above by consolidating the joint responsibility of the partners who contribute 
specific conditions to developing student learning over time. 


The model provides a way of describing the interrelationships among the 
dimensions. This description of the educational system and student achievement 
is not meant to predict future behavior, but rather to help in providing some 
understanding of the educational phenomena that can subsequently be acted 
upon to improve the situation. It is intended to permit multiple levels of analysis 
— province, community, school, classroom and individual students — so that a 
range of interpretations can be drawn and improvement targets proposed (Cziko, 
1989). 
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Education is a complex social enterprise. Focusing on schooling alone, without 
regard for the other influences that shape its direction, will likely not result in 
significant change. This model incorporates the responsibilities of the other major 
partners and recognizes conditions which contribute to student outcomes. For 
educational constituencies to determine whether or not students are making 
appropriate progress, it is necessary to examine results over time to determine 
what changes should and have been effected. 


The EQI initiative set three criteria for a successful indicator system: an 
interpretative framework to describe variation among students and schools; 
student outcomes related to the educational enterprise; and points of reference 
for comparing results (Alberta Education, 1988). 


¢ The interpretative framework consists of the partners (schooling, family and 
society) and conditions (context, inputs and processes). Together the partners 
and their respective conditions provide a framework for helping to understand 
the student outcomes which schools, the family and society strive to produce 
through adjusting the inputs and processes; the context is not as readily 
changeable. School jurisdictions and government collect a lot of information 
on context and inputs; some also collect information on instructional 
processes. School jurisdictions are interested in results for students in other 
jurisdictions of comparable size, location and economic base. 


¢ Student outcomes include a broad range of the behaviors desired by schools in 
the cognitive, affective, physical and social domains; the outcomes selected 
for measurement should relate to the goals and priorities of the local school 
jurisdiction and the province. 


¢ Points of reference include time, groups and targets. Assessment of selected 
indicators can occur either on an annual or a periodic basis; for example, 
Alberta achievement tests and diploma examinations are administered 
annually to students in grades 3, 6, 9 and 12, whereas assessments such 
as those conducted by the International Association for the Evaluation of 
Educational Achievement (IEA) and the International Assessment of 
Educational Progress (IAEP) occur periodically in selected subject areas. 
Groups for comparing results can include local, provincial, national and 
international counterparts. Targets include benchmarks and standards. 
Benchmarks describe the existing level of conditions and outcomes whereas 
standards define an optimal or desired level; targets specify a feasible level 
of improvement within a predetermined period of time. A target sets an 
improvement increment over what exists (the benchmark) in an attempt to 
reach what is desired (the standard). 


Design 


The Terms of Reference (November 1988) provided direction to the EQI projects. 
School jurisdictions that were already concerned with measuring educational 
quality were identified and invited to submit a proposal. Alberta Education 
provided the funds to assist these jurisdictions to improve their assessment 
procedures and to share the results with others in the province. 
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The 1988-89 school year was devoted to developing the conceptual framework 
and implementation plan, developing the Terms of Reference for the projects, 
initiating the collaborative action research projects, and organizing the 
committees. The projects began in 1989 and formally concluded in 1992. 


Coordination of the EQI initiative resided in the Policy and Planning Branch of 
Alberta Education. A provincial coordinator was responsible for the initiative. 
Two committees also provided assistance to EQI. The Working Group consisted 
of the liaison staff from each of the five Regional Offices of Education (located in 
Grande Prairie, Edmonton, Red Deer, Calgary and Lethbridge). This committee 
was responsible for monitoring the projects. It met on a regular basis to discuss 
progress, identify issues and strategies, and to plan the provincial meetings. The 
second committee was the interdivisional Advisory Group with representatives 
from branches within the Student Programs and Evaluation and Finance and 
Administration Divisions of Alberta Education; it met on a periodic basis and 
provided input and feedback. Both committees participated in the semiannual 
provincial meetings. 


Local Development 


Each project was autonomous and the responsibility of a local coordinator and 
project team. This coordinator reported to the superintendent for the school 
jurisdiction and was responsible for the developmental work and the management 
of the tasks identified in the contract between the school jurisdiction and the 
department. The contract required annual progress reports as well as a final report 
upon completion of the project in 1992. Each project was guided by a local 
Steering Committee consisting of the local coordinator and other staff from the 
participating school jurisdiction, liaison staff from the Regional Office of 
Education, and the provincial coordinator. The Steering Committee's 
responsibilities included consultation and evaluation of the terms of the contract. 


Each participating school jurisdiction was responsible for its own project because 
each represents a community which has values it wishes to impart to its students 
through its goals, priorities and expectations. It was important that each project 
reflect these aspirations by involving its public — students, teachers, administrators 
trustees, parents, others — in discussions on which indicators to include and why. 
Without input from the larger community, the indicators might not provide a 
representative picture of the expected outcomes of the educational system. 


Each participating school jurisdiction developed and implemented a local 
indicator system having three components: (1) a set of indicators including an 
interpretative framework, student outcomes and points of reference; (2) methods 
to collect, analyze and interpret the information; and (3) a reporting and 
dissemination strategy to inform diverse audiences of the results. The selected 
indicators represent each jurisdiction's goals and priorities and reflect the 
community's expectations. Every project had three phases, each of approximately 
one year's duration. The first year, 1989-90, was a developmental one and the 
subsequent two years resulted in field-testing the prototype sets of indicators and 
procedures, and then refining them. 


School jurisdictions, if they are to make sense of the outcomes of their educational 


enterprise, need to interpret the results in terms of their local circumstances. In 
the process of developing a preliminary set of indicators, trying it out in their own 
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jurisdiction and refining it on the basis of the results, jurisdictions took the 
opportunity to develop and use indicators that are flexible, responsive to local 
needs, have community support, and can provide them with the type of 
information they need for setting policies, practices and procedures. 


Provincial Support 


Alberta Education supported this initiative by providing resources (funds and 
consultation), information and coordination. Alberta Education committed $1 
million to the EQI initiative to support the three-year concurrent research projects. 


The Collaborator — This quarterly newsletter was introduced as a communications 
mechanism for the project teams and Alberta Education. The first issue was 
released in September 1989 at the first provincial meeting. It provided an 
overview of EQI and each of the ten concurrent collaborative action research 
projects. Eleven subsequent issues were published. Each June issue contained 
an annual status report. EQI participants regularly contributed material on the 
progress of their projects. This 92-page record documented events and activities 
over the three years. 


Technical Documents — Alberta Education developed three technical documents 
to support the EQI initiative. The first, an Annotated Bibliography, was initially 
distributed in July 1989. It was subsequently revised, expanded and reorganized 
and a second edition released in January 1990, followed by a supplement in 
December 1990. This second edition contains approximately 350 annotations 
organized into 14 themes under three major headings: indicator systems, 
interpretative framework and outcomes. It has been distributed to all school 
jurisdictions in Alberta to promote the discussion of results-based education. 

The second report, Methodological Considerations, consists of a rationale for 

a balanced approach to quantitative and qualitative analysis, followed by 
considerations in employing these two methods. The third document, Jnventory 
of Assessment Instruments, contains a description of technical criteria for 
analyzing instruments, and critiques of more than 90 instruments. The inventory 
identified potential assessment instruments and screened them according to their 
psychometric properties and suitability for EQI. It provided participants with a 
convenient pool of possible measures for the types of indicators that interested 
them. 


Semiannual Meetings — The key participants — Alberta Education staff, project 
coordinators and superintendents — met semiannually to share information, discuss 
and interpret findings, and identify issues and strategies. These meetings served 
to maintain personal contact among participants, to promote synergy, and 
encouraged people to meet deadlines. 


Professional Development — An important component of the EQI initiative was 
professional development. The November 1990 meeting featured concurrent 
workshops on the Inventory of Assessment Instruments and qualitative methods 
described in Methodological Considerations. These sessions provided participants 
with the opportunity to work through the material, ask questions, and discuss 
issues. The interactive computer version of the Annotated Bibliography was 

also demonstrated. The January 1992 meeting featured clarifying final report 
expectations and criteria for ensuring quality reports. There were also two 
concurrent workshops. The first featured data analysis and presentation using a 
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statistical and graphic utility; participants were able to try out StatView on five 
Macintosh microcomputers and a PowerBook. The second workshop focused 
on communicating with the public; it involved developing a communications 
strategy and group work in developing strategies for the projects. 


Technical Advice — In 1991, two measurement experts were engaged to provide 
technical advice to the project teams. University of Alberta professors Tom 
Maguire and Todd Rogers reviewed the materials prepared by the teams and, 
together with the provincial coordinator, visited each team to discuss the projects. 
Each team received written comments on its project and suggestions for 
improvement. This individualized assistance supplemented that provided 

by the provincial coordinator. 


The Projects 


Achieving Quality presents a summary of each of the ten projects that have 
devised many new indicators and practices for assessing the quality of education 
in their districts. The new knowledge generated by the projects can be used by 
others. The summaries have been organized into three major areas — system 
accountability, teaching and learning, and alternative student assessment 
strategies, each corresponding to the major focus of the projects. Within each 
area, there is diversity in types of indicators, approaches and reporting results. 


System Accountability 


Three projects focused on accountability at the district level. They developed 
indicator systems to examine the district as a whole, at the school level, and 
for specific programs. 


The Lacombe/Rocky Mountain Project developed a comprehensive school system 
review process encompassing student, teacher, program, school and school system 
evaluations. Its design consisted of five interrelated components: involvement of 
educational partners in developing and implementing the review, development of a 
set of goals of learning to form the basis of the review, identification of outcome, 
condition and process indicators, identification of ways of measuring the indicators 
and interpreting the results, and a planning process to use the findings. 


Grande Prairie Public developed school and district profiles to inform stakeholders 
of the quality of education in its district. The profiles provide information in four 
areas — student achievement, school climate, quality of instruction, and funding — 
and will be published annually. 


Lethbridge Public developed a collaborative model for school and program 
evaluation. The model includes inputs, student outcomes, and identifies 
assessment processes. It is generalizable and could be adapted for other 
programs. 
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Teaching and Learning 


Two projects focused on teaching and learning — one examined teaching behaviors 
and another outcome-based education. 


Spirit River identified 26 behaviors of effective teaching and grouped them into 
four areas — planning and preparation, instructional strategies, communication 
skills, and pupil/teacher relationships. In 1991, teachers were observed on these 
behaviors and a composite performance baseline prepared. Student perceptions, 
attitudes, achievement and discipline data were also collected. Data collection 
was repeated in 1992 to provide a basis for comparison. 


Fort McMurray Catholic implemented the principles of outcome-based education 
in mathematics from grades 3 to 10, and measured student success in achievement, 
responsibility and attitudes. 


Alternative Student Assessment Strategies 


Five projects examined alternative ways to measure student success. These 
included portfolio assessment and growth, exemplars of performance, responsible 
student behavior, social skill development, and affective outcomes. 


Edmonton Public developed mechanisms for student portfolio assessment and 
computer technology for communicating and storing information about student 
growth, and established and validated curriculum-based district level assessment 
strategies. 


Calgary Public and Catholic developed indicators of quality student performance 
in art and mathematics. EQI-Art focused on the product of learning, that is, the 
artwork prepared by the student, while EQI-Math focused on the process of 
student thinking in solving mathematical problems. 


Brooks explored ways to measure responsible student behavior. This project 
collected data from a variety of sources and involved the community in 
reaffirming its commitment to positive student behavior. 


Fort McMurray Public focused on desirable student social outcomes. The three- 
level approach to measuring social skills includes a diagnostic screen and a 
manual of activities to help students develop effective personal, interpersonal, 
social reasoning and classroom behaviors. 


Lethbridge Catholic developed a taxonomy of student affective behaviors in five 


areas — self-worth, interpersonal relationships, world awareness, learning, and 
spiritual life. 
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Overview 


This first chapter has introduced the Educational Quality Indicators initiative. It has 
described the background and rationale for this partnership with participating school 
jurisdictions. The conceptual framework guided the direction of the ten projects and 
provided a common framework and language for results-based education. Alberta 
Education supported the projects with resources (funds and consultation), information 
and coordination. 


Each project selected aspects of the indicator system based on local goals, priorities 
and expectations. The ten projects have been organized into three major areas 
dealing with system accountability, teaching and learning, and alternative student 
assessment strategies. The next ten chapters of Achieving Quality provide a summary 
of each project. A final chapter discusses EQI accomplishments and next steps. 
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School System Review: 


A Comprehensive Process 


County of Lacombe No. 14 and 
Rocky Mountain School Division No. 15 


The County of Lacombe and Rocky Mountain School Division initiated a study to 
develop and implement a comprehensive school system review based on quality 
indicators of student learning and indicators of related conditions and processes. 
A further purpose was to combine teacher, program, school and school system 
evaluations within the review process. The design of the project consisted of five 
interrelated components: involvement of educational partners in developing and 
implementing the review, development of a set of goals of learning to form the 
basis of the review, identification of outcome, condition and process indicators, 
identification of ways of measuring the indicators and interpreting the results, 

and a planning process to utilize the findings. 


The components of the review were collaboratively developed over a two-year 
period. A balanced set of goals in four groups — intellectual, social, personal 

and vocational learning — was developed and irdicators of cognitive, affective and 
behavioral learning were identified. Nine significant measures were identified or 
developed for the outcome indicators whereas four measures and six processes 
were selected for condition and process indicators. Locally developed instruments 
and other processes were field-tested. 


The full system review was implemented in the third year. Data were collected 
and have been assembled, and a procedures handbook on implementing the review 
has been completed. Individual teacher reports on classroom conditions and 
processes and school and systems reports, organized by the goals of learning, 
have also been finalized. 


The review process was found to be sound and feasible to implement. The findings 
on student learning reveal that a comprehensive description of student learning by a 
system of indicators is possible. 


To date the interpretation and planning processes have not been completed, but both 
boards are committed to using the results in a planning process followed by four 
years of implementing improvements. 


The review provides better data for accountability and more definitive data for 
planning. It balances evaluation, planning and implementation activities more 
appropriately and uses data for a variety of reports. The process could be adapted 
for use in other school systems to yield similar benefits. 
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Introduction 


The County of Lacombe No. 14 and the Rocky Mountain School Division No. 15, 
two small rural school jurisdictions in central Alberta, have had a continuing interest 
in being accountable and in improving the quality of education. Commencing in 
1985, both systems have conducted satisfaction surveys with their publics, reported 
the results and established improvement plans. Additionally, to adhere to 
departmental policy, the County of Lacombe developed an integrated teacher and 
school evaluation process whereas Rocky Mountain School Division concentrated 
on more traditional teacher, school and program evaluations. Both, as is usual, 
reviewed and reported on student achievement tests and diploma exams. Hence 
when the provincial initiative on educational quality indicators became a reality, they 
welcomed the opportunity to move to another step in the evolution of evaluation in 
their systems and to develop a comprehensive school system review process based 
on educational quality indicators that would combine all components of evaluation 
into one package. 


Rationale 


Although both school systems had been devoting an extensive amount of time to 
a variety of commendable evaluation activities, they did not see concomitant 
improvements being made in the educational program. Part of the reason for this 
lack of program improvement was that supervisory personnel and school staffs 
were devoting most of their time and effort to evaluation with little time left for 
developing and implementing follow-up improvements. The publics involved in 
the surveys and other evaluative activities were also concerned about a lack of 
benefits from these activities. System personnel were also cognizant of the many 
times when educational change in system decisions were precipitated by results on 
a single indicator or the occurrence of a single incident. 


System personnel, nevertheless, felt that many worthwhile evaluation activities 
had been developed in the systems: the public reporting of student results, the 
satisfaction surveys and the teacher, program and school evaluations. Thus, the 
systems wanted to address the concerns, utilizing existing evaluation activities but 
to evolve further by basing evaluations on educational quality indicators. They 
wanted to develop a more comprehensive evaluation system, one that was efficient 
and that included a process for planning and implementing improvements. 


Purposes 


The primary purpose of the study was to develop a comprehensive school review 
process based on student outcome indicators and the condition and process 
indicators that contribute to student learning. Thus, the review process would not 
only determine how well students were learning, but would also determine the 
conditions and processes that were contributing to learning and those that were 

not. The findings would be utilized to develop a plan for enhancing student learning 
and to inform the public about the state of learning in the system. 


Another purpose was to develop an efficient evaluation system that would maximize 
the use of the findings to generate teacher, school, program and school reports. 
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Design 


The school system review process is based on an educational quality indicator 
system that focuses on indicators of student outcomes as well as indicators of 
conditions and processes that affect student outcomes. A review of the literature 
suggests that an indicator system should: 


¢ be based on a comprehensive statement of goals of learning which are a local 
interpretation of provincial goals (David, 1986); 


¢ include multiple indicators that comprise a system to increase interpretative 
power (Goertz, 1989); 


¢ include a point of comparison such as a larger group or a previous result, 
and include conditions and processes to assist in interpreting outcomes (US 
Department of Education, 1985; Goertz, 1989; Haertel, Katzenmeyer, & 
Haertel, 1989); 


* focus on indicators that are enduring, easily understood and feasibly measured 
(Oakes, 1989); 


¢ produce information that can be readily used for planning improvements or 
making policy decisions (US Department of Education, 1985); 


* involve all educational partners in the development and implementation of the 
indicator system (David, 1986); 


¢ be designed as an integral part of the operation of the school system (Kaufman, 
1988). 


The design of the school system review process, incorporating the above 
characteristics of an indicator system, consists of five distinct but interrelated 
components: 


1. a plan for involvement of all educational partners in its development and 
implementation; 


2. acomprehensive statement of goals of learning; 


3. asystem of indicators to measure all goals of learning and the contributing 
conditions and processes; 


4. a process for measuring and interpreting learning; 


5. a planning process. 
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These components are shown in Figure 1 below and are described further following 
the diagram. 


Goals of Student Learning 


‘ Outcome, Condition and 
Educational / Process Indicators 


Partner n 
Interpreting Learning 


Figure 1: The Project Blueprint 


Educational Partner Involvement 


The educational partners were involved in the development and implementation of 
all components of the school system review process: the specification of goals, 
identification of indicators and comparators, the development of some of the 
measures, the interpretation of the results and the development of an improvement 
plan. There were generally four types of involvement by the educational partners. 
First, there was a steering committee, representing a broad cross section of the 
educational community, in each jurisdiction that planned and sanctioned directions 
for the project. Second, a process committee of parents, teachers and 
administrators in each system was involved in the detailed development and 
implementation of the review system. Third, there was direct and wide 
involvement of parents, staff and the public in some of the developments. 

Fourth, administrators and staff were used as a sounding board as the process 
was being developed. 
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Goals of Student Learning 


The development of a comprehensive statemen: of goals of student learning was 
required to provide a focus for the school system review. As a result, the 
evaluation could then be based on the specific results that each system wished to 
achieve on a local interpretation of provincial goals. To accomplish this statement 
of goals, the steering committee, through a group process and using assessment 
guidelines provided by the coordinator, reviewed a number of goal statements from 
the department. Although consensus was achieved at this stage on a goal statement 
in each jurisdiction, the statements were refined when the participants in future 
meetings were identifying outcome and process indicators. Goals were found to 
overlap and some goals were found not to be measurable. At this stage, it was also 
suggested that the goals be put into four groups: intellectual, social, personal and 
vocational. It was felt that this categorization would enhance understanding of 

the goals and promote the appropriate emphasis. Furthermore, the goals were 
categorized on the basis of responsibility for implementing them — primary for 
school responsibility and shared for both schocl and family responsibility. 


The significance of this exercise of developing a goal statement is that the 
jurisdictions now have a statement of goals of learning with a local flavor that 
educational partners understand and accept as a basis for educational programs 
and their evaluation. The goals that were agreed upon and sanctioned by the local 
steering committees are outlined in Figure 2. 


Outcome, Condition and Process Indicators and Measures 


Utilizing the statement of goals of student learning, indicators of achievement of the 
goals, appropriate existing or required measures and comparators were identified at 
meetings of parents and teachers at all schools in both systems. 


The meetings also succeeded in increasing awareness and support for the project as 
well as the use of multiple indicators since there was a general concern about using 
single indicators to describe the quality of education. 


Through the workshop sessions and subsequent revisions by project personnel and 
committees, outcome indicators relevant to any jurisdiction review were identified. 
The indicators were organized into three categories: cognitive, affective and 
behavioral. The cognitive indicators included achievement, program participation, 
completion rates, independent learning and problem solving; the affective indicators 
were student satisfaction, self-concept, attitudes, motivation and aspirations; the 
behavioral indicators were desirable characteristics, participation in related 
activities, community service, attendance, image, health and fitness. 


A sample of the outcome indicators, measures and comparators for the goal to 
"develop knowledge, skills and attitudes in language arts and communication" is 
contained in Figure 3. A similar outline of indicators, measures and comparators 
is available for every goal of learning. 
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Intellectual 
learning 


Social learning 


Personal learning 


Vocational 
learning 


Develop intellectual curiosity and a desire for independent lifelong 
learning. (shared) 


Develop knowledge, skills and positive attitudes in language and 
communication, mathematics, practical and fine arts, sciences and 
social studies. (primary) 


Develop ways of learning including: 

© skills of finding, comprehending, organizing, analyzing, applying and 
synthesizing information; 

¢ skills of learning through technology; 

* skills of studying. (primary) 


Develop a sense of community responsibility which embraces: 

* an understanding of and appropriate participation in citizenship at the 
local, national, and international levels; (primary) 

respect for law and authority, public and private property, and the 
opinions and rights of others; (shared) 

appreciation of the importance of tradition and culture; (shared) 

the careful use of natural resources and the preservation of the 
physical environment. (shared) 


Acquire knowledge and develop skills which contribute to physical 
well-being. (shared) 


Acquire knowledge, develop skills, attitudes, and habits that contribute 
to emotional well-being including: 

* achieving a positive self-concept; 

* acquiring a high level of self-discipline and individual responsibility; 

* acquiring an ability to respond to change; 

* developing short and long-term personal goals. (shared) 


Acquire knowledge and skills, attitudes, and habits for individuals to be 
successful in the world of work. (primary) 


Figure 2: Goals of Student Learning 
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Intellectual Learning: Develop knowledge, skills and positive attitudes in language and communication 


Outcomes __ Indicators Measures Comparators Grades 
cognitive achievement language arts achievement test provincial and system 3,6&9 
results 
English 30 and 33 diploma exams provincial and system 12 
results 
affective attitude School Subjects Attitude Scales test norms 6,9&12 
(SSAS) 
behavioral participation in document review system and provincial rates 3,6&9 
achievement tests 
participation in or the need document review, survey provincial rates 3,6&9 
to participate in remedial 
programs 
participation in grade 12 document review provincial and system rates 12 
diploma courses 
course completions document review provincial and system rates 3,6,9& 12 
amount of leisure reading student survey system average 6,9&12 
participation in related student survey system results 6,9 &12 
activities 
image - educational survey provincial and past survey N/A 


partner satisfaction 


results 


Figure 3: The Indicators, Measures, Comparators and 
Grades of Administration for One Goal 


The process committee consisting of teachers, administrators and parents with 


representation from each school was formed to identify context, input and process 


variables that affect student learning. By utilizing a number of system documents 
including the systems’ mission and improvement goals, previous inservice program 
agendas, school system program and school evaluations, as well as a document 

entitled What Works: Research about Teaching and Learning (US Department of 


Education, 1987), the committees were asked to identify context, input and process 
variables that were relevant in each system but were also identified in the literature 


as affecting achievement. 


There was some initial confusion about the distinction between context, input and 
process variables as well as a way of organizing the variables. As a result, the 
coordinator decided to use a categorization adapted from Oakes (1989) that 
identified three sets of conditions which affect student achievement: 


* access to learning includes those variables chat affect students’ opportunities 


to learn; 


¢ press for achievement includes those variables that encourage students to 
achieve better; and 
¢ professional teaching conditions are the variables that empower or constrain 
educators as they implement the instructional program. 
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It was felt that these three categories would be more readily understood by all 
educational partners. Furthermore, to avoid confusion it was decided to call them 
simply "conditions and processes" rather than context, input and process variables. 


Indicators of the above sets of conditions and processes were identified in four 
areas: the home, classroom, school, and school system. Figure 4 outlines the 
condition and process indicators along with the instruments or methods for 
measuring the indicators. 


Measuring and Interpreting Learning 


The identification of measures in Figures 3 and 4 above was guided by the 
following considerations: 


* existing measures such as achievement tests and diploma exams would be 
utilized as much as possible; 


* existing standardized measures would be used wherever possible to enhance 
reliability and validity; 


¢ the measuring devices used would be tests and inventories, surveys, document 
reviews, self-evaluation and observations. 


As aresult, the measurement of most indicators was accomplished by utilizing 
existing instruments and documents. Three locally developed instruments were 
used in the review: the Satisfaction Survey, the Adolescent Health Questionnaire 
and the Classroom Observation Scales. 


The instruments used to measure the outcome, condition and process indicators are 
described below, followed by a discussion of the organization of the data from all 
instruments to describe student learning and to prepare teacher, school, program 
and school system reports. 


Outcome Indicators — The indicators of student achievement in the four areas of 
learning — intellectual, social, personal and vocational — were measured by diploma 
exams, achievement tests, standardized tests and inventories, teacher student 
evaluation systems, locally developed surveys and inventories, and document 
reviews. The instruments and documents used to measure achievement of the 
indicators in the four areas of learning are briefly outlined below. 


In intellectual learning, student achievement in the core subjects was measured 

by the provincial achievement tests and the diploma exams. Achievement in non- 
diploma math and science courses, in fine and practical arts, and in work experience 
was measured by final marks awarded by classroom teachers. Achievement in 
intellectual curiosity was measured by the survey while achievement in ways of 
learning was measured by the achievement tests and diploma exams, portions of 
the Canadian Tests of Basic Skills (1982, 1989) and the Learning Preference 
Questionnaire (1987) which determines students' approach to learning. 
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Condition Area Indicator Measure 


access to schooling home family structure survey, document review 
socioeconomic status 

student mobility 

attitudes ard actions of importance to survey 


education 


press for achievement home 


access to learning classroom ¢ instructional presentation self-evaluation, observation, 
* classroom environment survey 
* academic engaged time 
* adaptive instruction 
¢ instructional planning 
* grouping practices 
¢ ability and nature of class 
* materials and resources 
* parental involvement 
press for achievement classroom * teacher expectations self-evaluation, observation, 
* relevant practice survey 
* informed feedback 
* progress evaluation 


* student recognition 


access to schooling school * location and size document review, survey, 
* course offerings committee study, external review 
* instructional time 
* teacher qualifications 
* co-curricular and peer programs 
* school grouping practices 
* parental involvement 
* curriculum articulation and resources 


press for achievement school * leadership School Effectiveness 
* climate ’ Opinionnaire, external validation, 
* academic focus survey 
* expectations 
* feedback 


* student participation 

* motivational strategies 

* mission 

* parental involvement 

* cosmetic care 

* type of homework 

* communication with public 


professional teaching school * class size document review, survey, 
conditions * teacher planning time School Effectiveness 
* teacher involvement in decision making Opinionnaire 


* teacher autonomy and flexibility 
* clerical and paraprofessional support 
* teacher attitude and morale 
access to schooling school system ~° location, size and assessment base document review, survey 
* public satisfaction with education 
* parental involvement in decision making 
* professional development 
* support for curriculum enhancement 
* mission, goals and actions for improvement document review, survey 
* high expectations for employee performance 
and studen: achievement 
recognition of employees and students 
monitor student achievement 
monitor instructional program 
communication with public 
staff involvement in system decision making document review, survey 
school flexibility and autonomy 


press for achievement school system 


professional teaching school system 
conditions 


eoeee ee e@ 


Figure 4: Indicators of Conditions and Processes Affecting Student Achievement 
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Participation rates for high school courses, the percentage of students taking each 
course, was calculated using data from two documents, the School Program Plan 
and the Summary of Enrollment by Age, Grade and Sex. Provincial rates were 
calculated by using data on the numbers of course participants and total grade 
enrollments provided by Alberta Education Information Services. Participation 
rates for students in core subjects in grades 3, 6 and 9 were calculated from the 
jurisdiction reports for achievement tests by dividing the number of students writing 
by the total grade enrollment. 


Completion rates for high school courses were determined by using information 
from the School Program Plan, the jurisdiction diploma exam report, and in the 
case of nondiploma exam courses, the summary reports of final marks. Completion 
rates for grades 3, 6 and 9 were calculated from the jurisdiction reports by dividing 
the number achieving the acceptable standard by the number eligible to write the 
test. 


Student attitudes in core subjects were measured by the School Subjects Attitude 
Scales (1983) while student and partner satisfaction with learning and student 
participation in related activities was measured by the Satisfaction Survey. 


In social learning, development of personal attributes and social skills, as well as 
involvement in promotional activities, was measured by the survey and review of 
documents. 


In personal learning, physical well-being was measured by participation in senior 
high physical education courses, by the Canada Fitness Award (1984), the 
Satisfaction Survey and through the Adolescent Health Questionnaire which 
measured four components of physical well-being: eating habits, involvement 

in activities, substance use and sexuality. 


The indicators of emotional well-being were measured by a standardized 
instrument, the survey, review of documents and the locally-developed health 
inventory. Student's self-concept was measured by the Piers-Harris Children's 
Self-Concept Scale (1984), a scale that measures overall concept as well as six 
subscales: behavior, intellectual and school status, physical appearance and 
attributes, anxiety, popularity, and happiness and satisfaction. Through the survey, 
student, parent and teacher satisfaction with student development of desirable 
characteristics, such as self-discipline and goal orientation, was measured. 


Condition and Process Indicators — As indicated in Figure 4, the condition and 
process indicators were measured by five means: the satisfaction survey, document 
review, a standardized instrument, the Classroom Observation Scales and 
committee study. Use is described in four areas: the home, classroom, school, 

and school system. 


In the home, the survey was used to measure family structure and attitudes whereas 
document review was utilized for socioeconomic status and student mobility. 


In the classroom, the survey was used to measure some indicators like grouping 
practices and student recognition. The locally developed Classroom Observation 
Scales were used for all indicators related to teaching and learning, such as 
instructional presentation and academic engaged time. The process for assessing 
these indicators involved teacher self-evaluation, observation and external 
evaluation by school administrators and external validation by system personnel. 
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In the school, using the School Program Plan, teacher and school timetables, and 
the report on teacher qualifications, indicators such as course offerings, instruction 
time, class size and teacher planning time were measured. The survey measured a 
number of indicators like grouping practices, parental involvement and teacher 
involvement in decision making. Through committee study followed by external 
review, curriculum articulation in the core subjects was measured. The School 
Effectiveness Opinionnaire (1989) measured ten school effectiveness indicators: 
leadership, academic focus, climate, expectations, feedback, student participation, 
motivational strategies, mission, parental involvement and cosmetic care. Following 
compilation of the opinionnaire results, an external validation team reviewed the 
data for a number of indicators. 


In the school system, a review of Education in Alberta: Facts and Figures 1991, 
the jurisdiction policy handbooks and system planning documents were used to 
assess a number of system indicators like assessment base, parent and staff 
involvement in decisions and support for curriculum enrichment. The survey was 
used to augment, confirm or dispute policy directions for the same indicators. 


Organization of the Findings for Reporting 


Since the focus of the school system review process was on student learning and 
the conditions and processes contributing to learning, the findings of the review 
have been organized and reported by student achievement of each goal followed by 
the relevant findings on the conditions and process. In this way, a comprehensive 
description, including overall achievement, achievement of major components, 
participation and completion rates, attitude, participation in related activities and 
student and partner satisfaction with learning is given for each goal. A sample of the 
description of student learning is included in the section on "Findings on Student 
Learning". 


Information on conditions and processes affecting learning in mathematics has not 
been included in this description since the assembly and assessment of the data by 
the process committees have not been completed at the time of writing. Determination 
of the conditions and processes affecting each goal will be decided by the collective 
judgment of the committee rather than by statistical means. 


Nature of Reports — Sufficient information was gathered through all of the 
measures to produce a variety of individual, program, school and jurisdiction 
reports. The relationship of the information gathered on all indicators to the various 
reports is as follows: 


1. at the classroom level, individual confidential teacher reports were written using 
the data on classroom indicators; 


2. atthe program level, core program reports can be written using the student 
outcome indicators, the aggregated information from the teacher evaluation 
reports and selected home and school indicators; 


3. at the school level, a school report was written on each goal of learning, 


including the information on all home and school indicators, student outcome 
and classroom indicators; 
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4. at the jurisdiction level, a system report was written using the information from 
all school system indicators as well as home, school, outcome and classroom 
indicators. 


The Interpretative Process — Following measurement of all indicators, student 
achievement on the goals of learning will be evaluated through an interpretative 
process. An interpretative team, representing all educational partners, will be 
formed for each domain of learning (intellectual, social, personal and vocational) 

in each jurisdiction. Each team's task will be to review the results of each goal of 
learning, and decide whether the results were excellent, very good, satisfactory 

or need improving. To make this evaluative judgment the team members will 
familiarize themselves with the instruments by completing each instrument, by prior 
establishment of desirable and acceptable levels of achievement, by determining the 
relationship of outcomes to the comparators and also the interrelationship of the 
indicators, and by determining the conditions and processes affecting the outcomes. 
Finally, after having made a judgment about the results on each goal, they will 
determine strengths and weaknesses and identify strategies for improvement. 


Following the interpretations in each domain of learning, the steering committee 
will review the results of each interpretation committee to insure consistency. 
Interpretative teams in each school will interpret school results through a similar 
process. 


A Plan for the Future 


To ensure that there is a follow-up to the evaluation, a modified strategic planning 
process has been incorporated as an integral part of the school system review. The 
general role of the planning team, representing all partners, will be to take the 
strengths and weaknesses identified in the interpretation process as well as other 
information and reorder curricular, human and physical resources to facilitate 
improvements in learning in weak areas and to maintain strong areas of learning. 
Specifically, the planning team will: 


¢ review the mission, beliefs and goals of learning; 


¢ review pertinent information internally and externally that impact on the system, 
including the results of the school system review; 


¢ identify critical issues and concerns; 
¢ establish improvement goals and strategies to achieve the goals. 
Following the completion of these activities, action plans to implement the strategies 


will be developed by committees representing all partners. At this point, board 
consideration of the plan including costing will occur. 
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The School System Review 


The study resulted in the development of a comprehensive school system review 
process based on student learning and the conditions and processes affecting 
learning. It has developed a way of measuring student learning with a system of 
indicators on a comprehensive statement of goals of learning, of measuring a full 
range of conditions and processes affecting student learning, and relating the 
conditions and processes to student learning. Sufficient data are produced by the 
various instruments and processes to generate individual confidential teacher 
reports, core program reports, school and school system reports. 


The review system consists of four related components: 


¢ ahandbook of procedures for implementing the school system review process; 

¢ supplementary handbooks on planning, communicating the results and 
providing inservice to staff; 

* reports on the detailed findings for a number of the instruments and procedures; 

* system and school reports on the findings of the school system review process. 


Each of these components is described forthwith. 


Handbook of Procedures for Implementing the School System Review Process 


This document is a complete guide for implementing the school system review. It 
describes the process including timelines and personnel responsibilities, provides a 
detailed description for measuring, analyzing and assessing the data for all of the 
indicators and gives the findings for each of the measures. The appendices provide 
a detailed guide to the administration of each instrument, as well as a copy of each 
instrument developed for this study. 


Overview of the Review Process — The handbook gives an overview of the review 
process and then provides a brief description of each of the components in the 
process: the goals of learning, the outcome incicators, measures and comparators, 
the condition and process indicators and their measures. The organization of 
findings by goal for a final report is described and the procedures for generating 
teacher, program, school and school system reports is given. Personnel 
responsibilities and timelines for administration are included. Finally, it describes 
a process for interpreting the results and developing a plan for the future. 


Procedures for Measuring, Analyzing and Assessing the Data for All Indicators — 
The largest part of the handbook is devoted to describing how each indicator is 
measured, how the information is analyzed and displayed, and how the data are 
assessed and summarized for the interpretation committee. Finally, a summary of 
the data that were collected on each of the indicators in 1991-92 is displayed. 


The Appendices — Further detail is given on the administration of the more complex 
measures in the review process. Generally, a document entitled "Guide to the 
Administration of the ..." is provided which gives the complete administration 
procedures for the instrument. Of particular interest in the appendix is a "Survey 
Resource Manual", developed by a reputable market consultant, which describes 
the appropriate methods for developing and administering a survey. 
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Supplementary Handbooks 


Supplementary handbooks have been prepared describing in detail the planning 
process, the communication plan and the staff inservice plan on the findings of 
the review process. 


Reports on the Detailed Findings for Some Instruments 


Separate reports provide the detailed findings for a number of the instruments 

acne in 1991-92. They include: 
School Effectiveness Opinionnaire (Renihan & Sackney, 1989); 

¢ Satisfaction Survey; 

¢ Learning Preference Questionnaire (Australian Council for Educational 
Research, 1987); 

¢ Piers-Harris Children’s Self-Concept Scales (Piers, 1984); 

¢ School Subjects Attitude Scales (Nyberg & Clarke, 1983) 

¢ Attitudes Toward the World of Work (Maguire, Romaniuk, & MacRury, 
1983). 


System and School Reports on the Findings of the School System Review Process 


The system findings on all of the indicators have been reorganized into a format 
based on each goal of learning. After a brief description of the school system 
review process, a full description of student learning with data from the appropriate 
instruments is given followed by the condition and process indicators that are 
contributing to learning and those that are not. In the discussion, the conclusion on 
student learning for the goal is given and plans for maintaining or strengthening 
learning are detailed. This report tends to be brief, relating all of the findings to one 
focal point, student learning, thus making sense of all of the data that have been 
collected. It is a comprehensive report to the public on the findings of the school 
system review. 


The findings for each school on all indicators were also organized by goals of 
learning and a report was written on the findings for each school. These reports are 
for the use of each school community. 


The Findings 


The study was designed to produce two kinds of findings: the feasibility of the 
school system review process, and the state of student learning and the contributing 
conditions and processes. The findings on the process have resulted from 
professional judgments as the process was developed and implemented, whereas 
the findings on student learning are the data produced by the various instruments. 
This section gives the findings on the feasibility of the process and the system 
findings on a sample of the goals of learning. 
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The Process 


Overall Design — The educational partners and the technical advisors supported the 
review process as a sound one including the overall design, based on the goals of 
student learning with a system of indicators of achievement, the instruments and 
processes, a way of interpreting the findings, and a planning process to utilize the 
findings. 


The Goals of Learning — The goals of learning (intellectual, social, personal and 
vocational) were found to provide an overall balanced statement of goals. Within a 
number of domains of learning, there was also a balance of goals that included the 
development of skills, knowledge and attitudes. 


Indicators, Measures and Comparators — In general it was found that there was a 
comprehensive set of indicators and appropriate measures and comparators to 
measure and interpret learning on most goals. There were some concerns with the 
measures for intellectual curiosity and the desire for lifelong learning, the measures 
for social learning and the comparators for learning in the intellectual domain. 
Recommendations for alleviating these concerns are made later in this summary. 


The findings on the usefulness of the major data collection instruments and 
procedures follow. 


The Satisfaction Survey produced data on a large number of indicators since the 
systems opted for a longer mail-out version rather than a shorter telephone survey. 
Its length and complexity meant that returns were slow and that the analysis took 
more time and resources than anticipated. In fact the extended analysis stretched the 
resources for the analysis of other instruments and processes. 


The School Effectiveness Opinionnaire produced interesting data on school 
indicators. Its administration and analysis were straightforward and efficient. 


Use of the diploma exam and achievement test jurisdiction reports, the School 
Program Plan, the Summary of Enrollment by Age, Grade and Sex as well as 
comparable data from Information Services, made it possible to calculate overall 
achievement, participation rates and completion rates for these subjects. 
Unfortunately, achievement on major course components of the diploma exams, 
such as the cognitive levels of knowledge, comprehension and higher level skills, 
cannot be determined in the same units as overall achievement. Overall achievement 
is given in percentage achieving the acceptable standard whereas achievement of 
major components is given as a raw score. The result is a loss in interpretative 
power. 


Student achievement in nondiploma exam courses was measured by analyzing 
summaries of teacher-awarded final marks. The collection procedures unfortunately 
were not as well defined as they should have been causing delays in receiving the 
data and considerably more effort in analyzing the results. 


The use of the Classroom Observation Scales to measure classroom indicators was 
conducted without difficulty. Although the coordinator had concerns initially about 
the time to carry out the evaluations, the school administrators readily accomplished 
the task since they would have devoted a similar amount of time to evaluation under 
"normal" circumstances. The inservice, structure and process were appreciated by 
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those involved in the evaluations. Use of self-evaluation by teachers, followed by 
external evaluation by school administrators and validation by system personnel 
provided the appropriate balance of participation from each level. 


All school staffs completed a review of the curriculum articulation and sequencing 
in each of the core subjects. The outlined procedures as well as the program review 
guides provided the necessary structure to complete the review. The validation by 
system personnel in one instance and regional office/system personnel in the other 
instance were able to validate the internal findings by using the outlined procedures. 


The implementation of the data collection procedures for all of the instruments in 
the same school year resulted in demands on classroom and personnel time. 
Fortunately each of the systems canceled all other evaluation activities for the 1991- 
92 school year making time available for implementing the review process. By 
using existing measures wherever possible and by administering the instruments to 
different grade levels, the time demands in any one set of classrooms were 
minimized. The impact on teacher time was found to be little different from the 
usual since surveys and program reviews had been the norm in both systems. 
Teacher involvement in self-evaluation did impact on teacher's time, but their 
appreciation in being more involved in evaluation more than compensated for the 
time spent. 


Interpreting Learning — The process of interpreting the findings whereby the 
interrelationship of indicators, the relationship of results to comparators and the 
contribution of conditions and processes to make a judgment about the value of the 
results was found to be an appropriate process. This was confirmed by the technical 
advisors. The process to date has not been implemented so it is not possible to 
comment on the workability of process. 


A Plan for the Future — This plan has been modeled after a traditional strategic 
planning process with the usual components. Overall the process is sound with 

an added advantage of being based on the strengths and weaknesses of student 
learning and the contributing conditions and processes. Thus, the plan for the future 
begins with the focal point of "student learning" and then identifies conditions and 
processes that need to be changed to improve achievement. The planning has not 
yet taken place so the feasibility of this process cannot be assessed. 


Findings on Student Learning 


To show the nature of the findings, student learning on a sample of goals is 
described. Goals have been selected where different measures have been used. 
Learning is described in mathematics, emotional well-being, and vocational 
learning. 


Unfortunately, the findings on the conditions and processes cannot be included 
because the process committees have not completed assessing the data for inclusion 
with the outcome data. 


Mathematics — Table 1 presents the indicators of student learning in grade 9 for 


the County of Lacombe. Learning is described by achievement, participation, 
completion, attitudes, participation in related activities and partner satisfaction. 
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Table 1 


Student Learning in Grade 9 Mathematics, 
County of Lacombe, 1991-92 


Grade Indicators Jurisdiction Province 
N (%) N (%) 
9 achievement | 242 27,888 
© total test 
- acceptable standard 64.5 67.4 
- standard of excellence 6.6 8.9 
e problem solving 
- acceptable standard 59.5 64.1 
- standard of excellence 8.3 10.7 
e subject matter 
- acceptable standard 76.4 75.6 
- standard of excellence 95 12.7 
participation 2 88.0 81.1 
completion 3 61.7 61.3 
attitude 4 244 
e evaluation scale mean 24.0 — 
e usefulness scale mean 32.9 
¢ difficulty scale mean 20.1 — 
participation in related activities 5 244 0.0 — 
educational partner satisfaction © 
e students 247 79.5 — 
* parents 113 77.2 — 
° teachers 71 70.6 — 


1 Achievement as determined by the June 1992 provincial achievement test where 85% of 
students are expected to achieve the acceptable standard and 15% are expected to achieve the 
standard of excellence. 

2 Participation is the percentage of students writing the test. 

3 Completion is the percentage of students eligible to write the test that achieved the acceptable 
standard. 

4 Attitude was measured by the School Subjects Attitude Scales at midyear; a mean of 24 
represents a neutral score whereas a mean above 24 represents a positive attitude toward a 
subject and one below 24 a negative attitude. 

5 Aper capita participation rate was determined by dividing the number of instances of 
participation by the grade enrollment. 

6 The percentage of partner satisfaction with learning mathematics was determined by the 
locally developed Satisfaction Survey. 
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Emotional Well-being — Table 2 presents the state of students’ emotional well-being 
in the County of Lacombe for grades 6, 9 and 12 in 1991-92. Emotional well-being 
is described by three indicators: self-concept, partner satisfaction and evidence of 
vandalism, suspension and expulsion. 


Table 2 


Student Emotional Well-Being, County of Lacombe 
Grades 6, 9 and 12, 1991-92 


Indicators Grade 6 Grade 9 Grade 12 
Female Male Female Male Female Male 
N=157 N=158 N=134 N=135 N=84 N=104 
(PR) _(PR) _(PR) _ (PR) __(PR)_ __ (PR) 


self-concept ! 


* total test 69 68 60 39 65 56 

¢ behavior 64 43 49 26 63 39 

* intellectual & school status 59 51 58 34 55 35 

¢ physical appearance 56 62 2 39 61 58 
& attributes 

* anxiety 55 66 50 51 51 60 

* popularity 30 39 44 26 46 44 


* happiness & satisfaction a7 60 53 40 60 54 


Students Parents Teachers 


Grade 6 9 12 6 9 12 6 9 12 
N 300 247 223 336 113 112 101 71 ~ 6421 


(%) (%) (%) 
satisfaction with learning 2 
+ self-discipline 722 89 #%7 88 64 #%71 84 62 «58 
* responsibility 78 90 88 83 77 7 8 6 6 
e pursuit of excellence 71 84 75 84 62 60 = 56 56 
* adaptability 69 91 80 = o91 79 7% 91 92 =«Oo71 

Gr. 1-6 Gr. 7-9 Gr. 10-12 

suspension & expulsion 3 5% 4.1% 1.0% 
vandalism 4 2% 


1 Percentile rank of group means for self-concept determined by the Piers-Harris Children's 
Self-Concept Scale. 

2 The percentage of satisfaction with learning desirable characteristics related to emotional well-being 
as determined by the locally developed Satisfaction Survey. 

3 The number of instances of suspensions and expulsions divided by the enrollment of each group. 

4 The number of instances of vandalism divided by the total enrollment in grades 1 to 12. 
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Table 3 
Vocational Learning in Rocky Mountain School Division, 
Grades 9 to 12, 1991-92 


Grades Indicators Jurisdiction Province 
(N=105) (N=11,499) 
% % 
10-12 work experience 
" achievement : 88.0 87.3 
e participation 13.1 10.1 
12 Students Parents Teachers 
(N=179) (N=73) (N=37) 
% % % 
partner satisfaction 4 
* prompt 81.0 73.6 57.9 
* neat 87.9 80.2 717.4 
¢ industrious 67.2 71.1 55.4 
* persevering 84.4 77.5 56.4 
* work habits, attitudes, skills and knowledge 76.3 69.7 57.6 
9 & 12 Grade 9 Grade 12 
Female Male Female Male 
(N=91) (N=104) (N=93) (N=79) 
(PR) (PR) (PR) (PR) 
attitudes 5 
* preparation 39 52 38 46 
* interest 34 43 20 27 
° diligence 36 38 44 36 
¢ laziness 64 69 57 50 
* job security 73 68 68 62 
* positive employer 52 51 38 32 
characteristics 
* independence 67 63 41 53 
* money 60 50 38 49 
* ambition 71 62 47 56 
* locus of control 75 71 71 67 
* confidence 28 39 23 22 
* negative employer 79 74 70 62 
characteristics 
* social relations 44 36 27 33 
* unemployment 66 60 76 66 
* general attitudes 35 58 60 46 


Percentage of students achieving 50% or higher in work experience. 

Percentage of enrolled high school students taking the course. 

Percentage of students that initially enrolled in work experience and achieved 50% or better. 
Percentage of satisfaction with learning desirable characteristics related to work as measured by 
the locally developed Satisfaction Survey. 

Percentile rank of group means for student attitudes determined by the Attitudes Toward the 
World of Work. 


hWNe 


nN 
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Vocational Learning — Table 3 presents student learning in the Rocky Mountain 
School Division for grades 9 to 12 in 1991-92. Vocational learning is described 
by achievement, participation and completion in work experience, by partner 
satisfaction and by attitudes to work. 


Conclusions 


The development and implementation of the school system review has resulted in 
significant conclusions. First, the process is pedagogically sound because the focal 
point of the review is student learning. Second, although most evaluation systems 
heretofore have focused on the process of education, the school system review 
more appropriately focuses on student learning and relates conditions and processes 
to achievement. 


Implementation of the school system review has been very demanding, but it 

was found to be feasible in terms of clarity of processes, time and personnel for 
implementation and the minimal time it takes from classroom learning. Although 
there were significant time demands on school administrators and staffs, the nature 
of the involvement resulted in a greater level of teamwork and enhanced sense of 
ownership of the evaluation and its results. 


The findings from the development and implementation of the process show that 
a comprehensive system of indicators to describe learning and the contributing 
processes is possible. The study has identified a significant set of indicators of 
learning and the conditions and processes affecting learning that are measurable 
and, when seen as a system, tell a comprehensive story about student learning. 


The review process was very effective in maximizing the use of the data to generate 
teacher, school, program and school system reports. 


Overall, the development and implementation of the school system review has been 
very successful resulting in a sound and feasible review process in the two 
jurisdictions. 


Implications 


The findings of this study have implications for education in general and specific 
implications for the two participating systems, other jurisdictions and the province. 


The design and findings clearly imply that the focus of an evaluation system in 
educational institutions should be on student learning. Although the "process" 
evaluations that have occurred in the past are designed to improve the process and 
hence learning, the time has come to relate process evaluations and student learning 
in a more direct way. 


Attempts to explain student achievement or the health of a school system or 
education in the province on the basis of one indicator are clearly inadequate. Use 
of other indicators along with achievement provides a better picture of achievement. 
The review process shows that it is possible to measure affective and behavioral 
goals and, therefore, this study has implications for school systems to evaluate 
more than cognitive goals. 
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Evaluation activities should be seen as a system of interrelated components which 
can produce a variety of outcomes. Typically, evaluations are one-purpose 
activities, for example, teacher evaluations, program evaluations, review of diploma 
results and satisfaction surveys. Not only is this an inefficient way of conducting 
evaluations, but it does not reveal the interrelationships of the findings from the 
various evaluation activities. Through the review process, a variety of reports can 
be produced simultaneously that show the relationship of conditions and processes 
to outcomes. 


The design and findings show an appropriate emphasis on evaluation, planning and 
implementing improvements. While the review process requires a concerted effort 
on evaluation for one year, it frees up four years for planning and implementing 
improvement. Overall it will mean that system and school administrators and staff 
will be able to devote more time to improving learning. These improvements will 
result from more comprehensive data and, therefore, will stand a better chance of 
improving learning. 


The review will provide more honest information for accounting to the public. 
Current practice of reporting one indicator presents only a part of the situation to the 
public. Reporting a system of indicators tells a more comprehensive story. 


For other school jurisdictions, a sound and feasible school system review process 
is available for use. It is not suggested that the full review be simply implemented 
as is. What is suggested is that the five components are integral to the process and 
should be a part of any review. If a jurisdiction wanted to implement a system 
review, it would be necessary to study each of the components to determine 
whether each would satisfy local needs. For example, the following questions 
could guide a review of the components. Are the goals of learning appropriate? 

Are indicators of learning appropriate? Are there better ways of measuring the 
indicators? In this way, the overall review process could be tailored to address local 
circumstances. 


For the province, the study shows that it is possible, even within current evaluation 
programs and information systems, to report on the health of the education system 
on a simple set of indicators. It is possible right now to report on participation and 
completion rates along with achievement on the diploma exams. It is also possible 
with data provided by Information Services to provide provincial results on 
achievement, participation and completion rates in all senior high courses. Using 
what is currently available, the province could provide better provincial data for its 
own planning and for public accountability. 


Recommendations 


To improve the school system review process, modification should be made to 
some of the instruments for measuring the indicators. 


¢ The satisfaction survey should be shortened and the information should be 
collected through a telephone survey. An investigation of the costs shows that a 
telephone survey is little if any more expensive and a considerably more 
efficient way of collecting data. 
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¢ Other ways of measuring intellectual curiosity and the desire for lifelong 
learning should be investigated. The three current indicators, participation in 
related activities, educational partner satisfaction and involvement of graduates 
in further education, are insufficient to describe learning. 


¢ The data collection and analysis procedures for learning in the nondiploma 
subjects should be further developed. 


¢ The technology use survey had to be postponed because of the strike. It should 
be developed for future use in school system reviews. 


¢ The use of the Fort McMurray Public District's Social Skills Diagnostic Screen 
and the processes for measuring social learning should be investigated. This 
would add a dimension of achievement in social learning to our current 
indicators of partner satisfaction, achievement of desirable characteristics and 
participation in related activities. 


¢ The use of exemplary pieces of student work (paintings, videotapes, writing) in 
various subjects should be investigated for use in interpreting student learning 
as recommended by the technical advisors. 


¢ The two school jurisdictions should monitor their activities in the four 
intervening years to maintain the integrity of the review process. 


It is recommended to Alberta Education that all achievement test and diploma exam 
results be reported in the jurisdiction summaries in the same units. If the results on 
the major components were reported in the same units as overall achievement, 
namely the percentage achieving the acceptable standard and the standard of 
excellence, the results would be more meaningful. 


Further it is recommended that Alberta Education make data available to calculate 
participation and completion rates. 


Lastly, it is recommended that Alberta Education report participation and completion 
rates in jurisdiction reports along with achievement on diploma exams, and finalize 
a set of provincial indicators as a result of the Educational Quality Indicators 
initiative. 


Follow-up 


The intent is to carry out the school system review process every five years with the 
intervening years being devoted to implementing the improvement plans. To insure 
that the integrity of the review process is maintained, the systems should monitor 
the system activities in the four intervening years to insure that the main thrust is 
improvement of learning and that "results on single indicators or single incidents" 
do not cause a reversion to former evaluation activities. The monitoring should, on 
the other hand, insure that required evaluations are identified and conducted. For 
example, in the intervening years it may be observed that student performance in 
mathematics has suddenly dropped. That would merit a review. Any such 
evaluations would be made on the basis of need rather than traditional scheduling 
of program or school evaluations. 
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Concluding Statement 


The study has resulted in the development of a comprehensive school system 
review that examines the full operation of a school system in one process. 
Accountability and general public support for education have the potential to 
improve as a result of reporting the comprehensive findings and improvement 
plans. The interest of parents in the school systems has been heightened because of 
their involvement in the project and growth in their knowledge and contribution to 
education has occurred. With its focus on student learning and related conditions 
and processes, the review is a powerful tool to describe the state of student 
learning, to identify strengths and weaknesses, and through planning and the 
luxury of time, to make improvements in learning. 
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District and School Profiles 


for Quality Education 


Grande Prairie School District No. 2357 


The Grande Prairie Educational Quality Indicators Project developed prototype 
annual school and district profiles to provide interested stakeholders with current 
and valuable information to judge the health of their educational system and to 
identify areas needing attention. Annual feedback allows schools to become more 
efficient and effective in delivering education to their clients. The report outlines 
the preliminary work in involving educational stakeholders to identify indicators 
of educational quality within the school district and describes methods to measure 
and report information in a manner that is meaningful and useful to stakeholders. 
The benefits of gathering and using indicators to assess educational quality and to 
plan educational improvements is also discussed. 


Rationale 


The Grande Prairie Public School District became involved with the three-year 
Educational Quality Indicators initiative specifically to meet four educational goals. 
First, the district was interested in developing a set of indicators of educational 
quality that reflect the views of the local educational stakeholders. Second, the 
district wanted to report the comparative results of the local indicators in an annual 
document which would reflect the quality of education within our district. The third 
goal was the development of a process to use annual reports to plan educational 
improvement within the district. The final goal was to test the indicators and 
processes in another school district to test their transferability. 


Purpose 


The Educational Quality Indicators Project was designed to identify, measure and 
report on indicators of quality education as defined by our educational stakeholders 
in the Grande Prairie Public School District. During the three years of the project, 
contact was made with approximately 4,000 educational stakeholders in the city. 
The contact made with the stakeholders provided the project team with the 
indicators used and also the level of stakeholder satisfaction with these indicators. 
The project developed measures for all the identified indicators that span the 
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cognitive, affective and behavioral domains. The annual reports on the indicators 
and their measures will provide stakeholders with historical data on which to assess 
and improve education in the city schools. 


Two assumptions were made at the onset of the project. The first was that 
educational stakeholders wanted and needed data which they could understand and 
which they agreed reflected the educational health of the schools and the school 
district. During the interview process in the first year of the study, the project team 
found this assumption to be correct. All stakeholder groups expressed confusion 
and frustration with the use of standardized tests as the main indicator of student 
success in the schools. They expressed a strong desire to use other measures 
which more clearly reflected the success of the schools and their students. 


The second assumption was that the existing stakeholders did not have a clear 
understanding of the present educational system and how it performed. This 
assumption also proved to be somewhat true when the team interviewed 
stakeholders who did not have direct contact with the system. The general public 
and parents seemed to base many of their views on the system they encountered 
when they were students in school. It was also apparent to the project team that 
employees and students did not have an understanding of the educational system 
or product beyond their own school building. 


Design 


The Grande Prairie Project was designed in three phases, one for each of its 

three years. During the first phase, the project team conducted interviews with 
representative groups to establish indicators of quality education which would 
serve as the basis for the annual profiles. The interviews were conducted using the 
nominal group technique and resulted in the team making contact with more than 
1,900 stakeholders. The team also conducted an extensive review of the literature 
related to effective schools and quality education. The final objective of the first 
phase was to have a workable set of definitions and indicators to continue into the 
second phase of the project. 


Phase two was dedicated to developing workable criteria for measuring the 
indicators identified during phase one. The initial task for the team was to cluster 
the indicators under larger indicator titles. The final result of the clustering process 
was the establishment of four areas: student achievement, school climate, quality 
of instruction, and funding. The second task for the team was to develop a measure 
for each indicator and to explore different methods to collect information on the 
indicators. The stakeholder satisfaction survey was also developed during phase 
two as an instrument to contribute to many of the indicators identified. 


The final phase of the project was designed specifically to develop a method of 
reporting to stakeholders on the identified indicators. This phase also resulted in 
work with the district and the schools to develop effective methods to gather and 
report on the data required. The result of the final phase saw the production of 
school and district profiles in our pilot schools and in the district. 
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School and District Profiles 


Table 1 presents the quality indicators, how they are measured, and their sources. 
Student achievement includes performance on provincial achievement tests and 
diploma exams, success of graduates, fitness, teacher marks, affective education, 
retention and attendance. School climate includes student and staff morale, 
turnover, behavior, and dropouts. Quality of instruction includes professional 
attributes, teacher inservice, instructional methods and expectations. Funding 
includes cost efficiency and staffing. 


The project produced three major products. The first was the stakeholder 
satisfaction surveys. These questionnaires were designed to obtain feedback on the 
four main indicator groups. Four types of stakeholders were surveyed — parents/ 
guardians, students, district staff, and the public. The questionnaires will be 
evaluated at the end of each year to determine what changes need to be made for the 
next survey. The surveys were delivered to the schools where participants were 
asked to record their answers on a scantron computer score sheet. This method of 
collecting and recording the stakeholder responses was determined by the team to 
be very successful. Two groups of stakeholders did not fill out the computer score 
sheet directly. The general public was contacted by trained student operators and 
their responses recorded on the computer sheets. The parents of the Composite 
High School were also contacted by a trained survey operator. Students of the 
other pilot schools were asked to deliver their parent surveys directly and all survey 
sheets returned to the school in good condition allowed the student's name to be 
entered into a cash draw. This format for returns was very successful at the junior 
high school level. Another product from the surveys was the adaptation of existing 
computer software packages to produce the graphed results of each question in the 
survey. 


Table 2 presents the stakeholder responses to the May 1991 survey. Annual survey 
results are an integral part of school improvement planning. Survey results flag 
areas that may become the focus for annual school or district improvement plans. 
Once an area is flagged and investigated using other indicators, it may become the 
target for improvement. The surveys also provide feedback on the results of the 
improvement action plan. 


The second major product was the individual school profile. These profiles serve 
as the communication link between the stakeholders and the school on the quality 

of education within the school and the actions schools are taking to maintain or 
improve that quality. The school profiles include reports on indicators requested by 
the school district as well as indicators judged valuable by site teams. A subsequent 
product from the school profile will be the development of a software package that 
will allow schools to produce profiles with minimum labor and low cost. 


The third product was the comprehensive school and district profile. This profile 
reported all the indicators identified by the stakeholders to reflect a quality 
education. Each indicator has a method of collecting and reporting representative 
data. A major software package is being developed to allow schools and the school 
district to report on all indicators in an efficient and effective manner. 
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Table 1 


Quality Indicators 


Indicator Measures Sources 
Student Achievement 
provincial achievement tests grades 3, 6 & 9 students 
diploma exams grade 12 students 
post graduate success graduate questionnaire 
Canada Fitness Award medals count 
teacher assigned marks marks table 
affective education programs, awards, participation 
retention rates district statistics 
attendance district statistics 
School Climate 
climate questionnaire 
student morale questionnaire 
staff morale questionnaire 
turnover district statistics 
behavior district statistics 
attendance district statistics 
lates district statistics 
expulsions district statistics 
dropouts district statistics 
Quality of Instruction 
professional attributes personnel records 
inservice budget expenditures 
substitute days school reports 
instructional methods site administrator 
expectations school/district policy 
monitoring site administrator 
curriculum administrator 
Funding 
cost efficiency secretary-treasurer 
staffing personnel officer 
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Table 2 
Grande Prairie Stakeholder Satisfaction in May 1991 (Percent) 


Indicator Parents Elementary Secondary Staff Public 
Students Students 
(N=456) (N=531) (N=571)  (N=253) (N=176) 


Student Achievement 


Satisfied 66 76 54 75 55 
Dissatisfied 12 14 29 12 18 
Need Information 14 9 14 8 26 
Unimportant 0 0 0 0 1 
No Response 8 2 a 6 1 


School Climate 


Satisfied 73 74 59 62 57 
Dissatisfied 10 14 23 25 18 
Need Information 9 9 12 6 ol 
Unimportant 0 0 0 1 2 
No Response 8 3 6 6 1 


Quality of Instruction 


Satisfied 67 75 62 62 23 
Dissatisfied 9 13 i 26 15 
Need Information 16 10 11 . 29 
Unimportant 0 0 0 1 1 
No Response 8 2 0 6 2 
Funding 

Satisfied 69 65 69 48 76 
Dissatisfied 8 22 24 35 8 
Need Information 12 10 | 8 13 
Unimportant 1 0 0 Fs 3 
No Response 10 3 0 7 0 


Note. Percentages may not total 100 due to rounding. 
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The products developed are invaluable when used to accomplish the intent of the 
project — to provide stakeholders with quality information on which to assess and 
improve the quality of education in the school district. The ultimate product of the 
project is the action plan. The plan for using the products of the project is a four- 
step process. The first step involves the formation of site-based improvement 
teams composed of stakeholders who assess the information available to them in the 
comprehensive school profiles. This site-based team then targets areas to improve 
for the upcoming school year. Once the areas of interest have been targeted, the 
second step of the process is initiated. This step involves the formation of an 
action team to develop a strategy that involves an action plan, timelines and an 
evaluation method to address the area of concern. Once the action plan has been 
developed, the third step of the process is to formally implement the action plan. 
The fourth and final step is to assess the results of the improvement action based 
on the indicators developed from the project. The final assessment may result in 
abandonment of the action plan if the improvement measure is negligible or 
inefficient due to labor requirements to affect acceptable change, or altering the 
existing action plan to provide more effective results or, finally, celebrating the 
success of the action plan. 


The format for using the products will provide each school and the school district 
with the ability to address areas of need specific to their own environment. It is 
expected, however, the school district will require all schools to report on common 
indicators of educational quality. 


Conclusions 


The implementation of an action plan involved the Grande Prairie Public School 
District and the County of Grande Prairie. Each district provided volunteer pilot 
schools to begin the development of school profiles. Once identified, our project 
team members informed the school administrators of the collection, presentation 
and process involved in the development of school profiles. The comprehensive 
school and district profiles contain the results of all annual data collection and are 
accessible to all stakeholders in the school district upon request. The format of the 
district report does not specifically identify individual schools but blend the results 
into a district profile. Public profiles will be condensed versions of the 
comprehensive school and district profiles. The public profiles will use short 
descriptions, charts and graphs to communicate on various quality indicators. 


Individual school profiles report on specific areas as directed by the superintendent 
and also reflect the needs of the individual school community resulting in different 
profiles for individual schools. The quintessential objective of the school profile is 
to provide school stakeholders with information that is meaningful and useful to 
them in planning school improvement and measuring the educational health of their 
schools. This objective is the driving force that necessitates collaboration among 
stakeholders to monitor improvement projects and to initiate required changes. 
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Implications 


The Grande Prairie Quality EQI Project initiated a process through which the health 
and direction of education became the communication of the performance of 
education in the district to stakeholders. The satisfaction survey results emphasized 
the need of the public and parents for more information on many of the identified 
indicators. The collection of the data will become much more efficient within the 
schools and more centralized at the district office. The use of computer technology 
to collect and report on the data will provide all stakeholders with information that 
can be easily accessed and properly interpreted and used in assessment and 
improvement of the district. This electronic data gathering and storing method 

has put the Grande Prairie School District in an excellent position to participate 

in the electronic information exchange advocated by Alberta Education. 


Recommendations 


The sample of the general public in the project did not include a wide variety of 
business and industrial representatives. Future expansion or replication of the 
study should include representatives of post-secondary educational institutions 

and the general public. The satisfaction survey worked well for the initial gathering 
of information on the identified indicators. The project team recommends the 
development of a validation process of each of the questions on the questionnaires 
over time to further enhance the instruments. Finally, a strong recommendation is 
given to the ongoing annual collection and publishing of educational indicators for 
the purpose of improving the quality of education in the school district. 


Follow-up — Ongoing Project 


The EQI Project is simply the start of a major effort within our district. Much work 
is ahead for stakeholders to use the indicators developed during the study to plan 
and monitor educational quality. As in any major educational thrust, stakeholders 
must be reinforced and encouraged to persevere. It is incumbent on the 
administration and the board of the school district to publicly maintain support 

for the direction of this project toward quality education. 


-The Grande Prairie Public School District is committed to the development of 
profiles for ongoing use in the district. The profiles will be adjusted to reflect 

the thirteen priority directions outlined by Alberta Education in Achieving the 

vision 199] report. The Board of Trustees of the Grande Prairie Public District 
unanimously passed a motion at the December 8, 1992 meeting which directed each 
school to produce annual profiles commencing in the 1992 -1993 school year. 
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Concluding Statement 


The initial efforts required to develop an indicator collection and reporting process 
within a school district seem overwhelming. The process must begin by developing 
a breakdown of the steps required to reach the goal of the project. Once these steps 
are separated, action plans can be developed to gain numerous small victories in the 
progress toward the overall objective of developing school improvements through 
the collection and reporting of educational indicators. 


The production of annual school and district profiles will be a continuous program 
in the Grande Prairie Public School District. The instruments will provide interested 
stakeholders with current and valuable data to not only judge the health of their 
educational system, but also to identify specific areas in need of attention. Annual 
feedback allows the schools to measure progress or the effect of efforts to improve 
areas of education. If the feedback indicates the efforts have not produced the 
expected or desired results, school teams may decide to abandon or change their 
action plans. Feedback enables schools and school districts to become more 
efficient and effective in delivering education to their clients. A more effective and 
efficient educational system, as identified by stakeholders, will result in a positive 
educational experience for all. . 
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A Collaborative Model for School 


and Program Evaluation 


Lethbridge School District No. 51 


In recent years there has been a marked increase in the emphasis and expectations 
for school and program evaluations. Throughout much of Alberta, the approach 
used has been predominantly a "top down" process with the planning and 
procedures carried out almost exclusively by personnel from Alberta Education 

or district central offices with assistance by external resource persons. A meta- 
evaluation, conducted by Lethbridge School District No. 51, concluded that limited 
changes were being realized by the evaluations. Teachers and school-based 
administrators were not actively involved and hence gained little from the exercise. 
A study of the related literature and procedures used in other locations led to the 
development of a "collaborative model" to empower school-based personnel to be 
actively and professionally involved in their school and program evaluations. The 
use of "educational quality indicators" served as a key component of the model. 


Rationale 


The evaluation of teaching and of teachers has received in-depth study but little 
effort seems to have been put into whether the models and procedures used for the 
relatively new focus on school and program evaluations are consistent with the 
findings of educational research. Because the stated goal of evaluation is almost 
always school improvement, there is a need to consider and utilize the research on 
quality indicators, school effectiveness, and school improvement. A collaborative 
model would ensure a "buy in" by those involved. This is essential as evaluation 
alone does not cause improvement. 


In the province of Alberta, as in other Canadian provinces and a number of 
American states, there has been an increasing emphasis and expectation from 
governments that school jurisdictions will develop and carry out formalized 
procedures for evaluation, including the evaluation of students, teachers, 
programs, schools, and school systems. 


The Lethbridge model involved the use of a large team (up to 55 persons) from 

the school district central office, other schools, Alberta Education offices, and the 
University of Lethbridge, spending up to three weeks of time on site for the school 
evaluation process. A common concern was whether the model (which was 
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extremely expensive in terms of professional time and expertise) was producing 
payoffs in professional growth and development. Perhaps the most significant 
concern was with the problem of implementing the recommendations emanating 
from the evaluation report. 


Administrators and boards need assurance that their models of school and program 
evaluation are consistent with increased effectiveness and school improvement as 

a result of the investments of professional time, expertise, and budget dollars. 
Stufflebeam and Shinkfield (1985) affirmed that if evaluations are to be useful and 
provide proper direction and guidance, "the evaluations themselves must be sound" 
(p. 183). 


Purpose 


The purpose of the Lethbridge School District No. 51 project was to develop a 
more effective model for school and program evaluation. Consequently, the 
following research questions were addressed. 


1. Is the current Lethbridge School District No. 51 model for school and program 
evaluation consistent with the literature on indicators of effectiveness, quality, 
and school improvement? 


2. Are the procedures, instruments, and data used in the Lethbridge School District 
No. 51 model valid and reliable? 


3. Does the current literature suggest characteristics or criteria of more effective 
and efficient models for school and program evaluation, in terms of indicators 
of effectiveness, quality, and improvement? 


4. If the findings for the above indicate "yes", can these findings be applied to the 
development of a collaborative model for a school system evaluation of the art 
and library programs? 


Design 


Warnica (1989) completed a case study of the Lethbridge model for school 
and program evaluation. It was primarily qualitative in nature, but included 
some correlations. The major areas of educational literature reviewed as a 
basis for this study and the proposed model were: school and program 
evaluation — theory and practice; effective schools research; indicators 

of effectiveness or quality; and, evaluation models, criteria, and standards. 


After his review of the literature and based on his career experience, Warnica 
(1990) concluded that the decision on the debate related to the external versus 
internal evaluation model falls clearly on the side of the school-based model. The 
convincing arguments of Boud and Donovan (1982), Common (1987), Herman 
(1986), Toffler (1980), Eisner (1985) and others provide sound reasons for a 
model which involves more active and professional participation of school-based 
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personnel. The empowerment of school-based teachers and administrators, and the 
school as the focus of action and development offer the greatest potential for real 
growth and development of the school, its staff, and its programs for students. 


Although the intent of the original research was to focus on school evaluation, 
senior administrators and trustees directed the move toward program evaluation. 
Using the study as a basis of information, work then proceeded on the application 
of this research to the development of a collaborative model for art and library 
program evaluations. These programs were selected for a number of reasons: 


* asystem perceived need to evaluate the effectiveness of these programs; 

¢ availability of central office leadership; and 

* concerns about the lack of expertise of the generalist teacher in the area of art 
evaluation as well as the lack of available research on this topic. 


Acting upon recommendations from the administration, the Board approved the 
development and implementation of a new evaluation format. Steering committees, 
consisting of generalist and specialist teachers representing elementary and 
secondary schools, administration, and Alberta Education personnel, were 
organized for each program area. The committees were given instruction and 
direction on the development of quality indicators. Both committees proceeded 
through the steps outlined in the art program flow chart (Figure 1). 


During the second year of the project, a model was developed and field-tested 
for evaluating the two identified programs. The collaborative model provided 

the structure for application of the effective schools research about evaluation and 
the use of indicators of effectiveness or program quality. Key components in the 
development of the educational quality indicators included the district-based staff, 
the community and, the resulting determination of comparators, standards, and 
quality outcomes for all programs. 


Following the collection of data through interviews and the administration of 
questionnaires to the various stakeholder groups, analyses were conducted. 

As a result of the findings, a system- and school-based action plan was developed 
and initiated. 


As a result of staffing changes, identified system needs and the availability of time, 
the art program became the focus of the project in the third year. Work continued 
through the various steps outlined: the identification and verification of the student 
outcomes; and, the identification, development, and refinement of evaluation tools 
and processes. System and teacher inputs were reexamined and the instruments 
modified as required. 


Findings 


The findings of the case study indicated that a new model for school and program 
evaluation should be developed. The result of che research was the development 
of a collaborative model which places much greater responsibility and control in 
the hands of the stakeholders — school-based administrators, staff, students, and 
parents. They became major participants and decision makers in school and 
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Figure 1: Art Program Evaluation Flowchart 
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program evaluations, in identifying areas of program and professional growth, 
and in bringing about change. The model rests upon certain basic assumptions 
grounded in the literature on school effectiveness and school improvement. The 
school is the primary unit of decision making (Smith & Purkey, 1985). If changes 
are to occur, they require ownership that comes from the opportunity to participate 
in defining change, and the flexibility to adapt it to individual circumstances. 
Change does not come from externally imposed procedures (Fullan, 1982). 


Schools need to identify areas requiring improvement and work actively toward 
this end. Empowerment of school-based staff and administration is fundamental 
to reviewing and improving schools. As a perception check, a follow-up 
questionnaire was distributed to the stakeholders at the conclusion of the art 
evaluation. The results from this questionnaire, in conjunction with a review 

of the content and the initiation of the program action plans developed by the 
schools, confirmed that the collaborative model was preferred. 


The collaborative nature of this approach resulted in a "buy in" by stakeholder 
groups. The development and use of quality indicators was positively received 
and their use incorporated into assessment procedures. 


As predicted by the art steering committee, evaluation procedures were identified 
as an area of concern. A third of the over 3,000 students surveyed did not 
understand how their report card mark was established; only 7% indicated a clear 
understanding of the process. Most parents (80%) were not aware of how their 
child was being evaluated and 37% of the teachers questioned indicated problems 
with evaluation procedures. A follow-up questionnaire addressed to teachers and 
administrators confirmed that through the application of the collaborative model 
and the identification of quality indicators, a positive change in attitude regarding 
assessment in art had taken place. 


The model presented in Figure 2, although originally developed for an art program, 
is basically a generic model that can be applied to other subject areas with few 
modifications. Assessment procedures to measure the desired student outcomes 
can be identified and/or developed. 


The Art Program Model 


Figure 3 presents the Results-based Art Program Plan, which is a refinement of 
Figure 2. It does not include the inputs of the home, community, and society, 
some of which are inherent under other headings. This model encourages the 
teacher to identify his/her individual mission statement/belief. It stresses the 
importance of inputs into the program; the need to take students from where they 
are with the preassessment and to track their growth along the continuum. The 
plan assists teachers in focusing on the desired student outcomes and, through the 
use of a variety of assessment procedures, determine the actual student outcomes. 


Model for Evaluation 47 Lethbridge Public 


uonenyeag J0J }2PoW 


@ Is committed and enthusiastic 


student needs and interests* 
@ Exercises efficient classroom 

management, organization* 
@ Makes quality use of time 


@ Estabishes positive climate- 


8Y 


repport* : ; 
@ Uses variety instructional 


@ Makes practical use of resources 


parents, staff 
@ Promotes staff interaction/ 


cooperation 
@ Participates in regulars PD 


* Indicators that are partially validated by questionnaires 
BWhvjn 


orang espuqueT 


Hon and enthusiasm for importance of 


@ Knowledge 
* (1, 2,3, 4,5, 6) 

@ Organizatonal skills to achieve objectives 
(1, 2, 3, 5) 

@ Critical thinking skits (2, 3, 5, 6) 

e@ Commament to task: self-directed nsk taker 
* (1, 2,3) 

@ Desire and respect for quality work (own & 
others)* (1, 2, 3, 5) 

@ Awareness of art in life/communiy — other 
cumures (1, 2, 3, 4) 

@ Ability to communicate visually and verbally 
(1, 2, 3, 5, 6) 

@ Self-esteem, self-respect, seifconfidence 
*(1, 3, 4) 

@ Creativity, originality, uniqueness of product 
(1, 2, 3, 5, 6) 


Figure 2: A Collaborative Model for Art Program Evaluation 
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Figure 3: Results-Based Art Program Plan 


Implications 


. Based on the current literature and the practical application, it has been 
demonstrated that a more effective model for school and program evaluation 
involving collaboration and the identification of educational quality indicators 
can be designed and implemented. 


. The development of a clear set of standards or indicators of educational quality 
would be a desirable step in moving the evaluation model on to more objective 
ground, and would allow much more professional involvement of school level 
personnel. 


. Although the art program evaluation was a practical application of the research, 
the model developed could be applied, with few modifications, to other subject 
areas. However, assessment procedures would have to be identified/developed 
for the different programs. 


. Community/stakeholder involvement in the evaluation process has proven to be 
very valuable input and should continue to be utilized. 


Recommendations 


. School and/or program evaluations should be designed using a collaborative 
model. 


. School and/or program evaluations should include the development and use of 
educational quality indicators. A vital component is the involvement of 
stakeholders in defining what constitutes quality. 


. School and program evaluation should be an ongoing process and not simply 
an event which takes place once every five years or so. An evaluation model 
should ensure that program and school evaluation becomes a natural and vital 
part of the curriculum and instruction cycle. 


. The program evaluation model that has been developed should be applied to 
other program areas in Lethbridge School District No. 51 and is recommended 
for other jurisdictions. 


. The results of this project should be disseminated to other jurisdictions by 
Alberta Education. 


. Further development and validation of the art program evaluation model should 
continue, particularly with regard to the refinement of the educational quality 
indicators and the verification of student outcomes through assessment 
procedures. 


. Teacher training in the use of a variety of assessment procedures that can be 
employed in measuring whether students have achieved the desired outcomes is 
essential. 
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Follow-up 


Lethbridge School District No. 51 is continuing to expand the use of this model to 
other program areas. Revisions/refinements to the collaborative model will be an 
ongoing process. The identification and development of assessment procedures 
for the measurement of student outcomes will be expanded to other programs. 


Interest has been expressed by the Calgary Public and Catholic EQI Project 
members to work cooperatively with Lethbridge to merge the two projects. 
This possibility will be explored. 


Closing Statement 


As a result of the research project, it has been determined that the former model of 
_ program and school evaluation was not consistent with the literature on indicators 

of effectiveness, quality and school improvement. New evaluation models should 

move away from the heavy emphasis on a "top down" approach to encourage and 

empower school staff and administrators to be actively involved and interacting 

in assessing the effectiveness of their own programs and in planning their 

improvement. A collaborative model for school/program evaluations in which 

the use of "educational quality indicators" is a key component, is more effective. 
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A Collaborative Teacher Evaluation Model: 


Teachers Do Make a Difference 


Spirit River School Division No. 47 


The Spirit River School Division Educational Quality Indicators Project involved 
the development and validation of a set of indicators to assess teacher effectiveness 
in a formative process. A major component of this project was the process. The 
extensive involvement of teachers in the formative evaluation aspects of the project 
impacted upon teacher effectiveness in the classroom and student outcomes. We 
intend to continue this project for three more years to determine its potential for 
export to other school jurisdictions within Alberta and elsewhere. 


Introduction 
Rationale 


We believe that teachers do make a difference, for as Broudy (1975) stated: "The 
teacher is still the key to schooling: with good teaching, almost any curriculum, 
school organization, and administrative invention seems to succeed" (p. 64). 
Experience and the literature clearly suggest that teacher and administrator 
involvement in the selection of educational indicators was critical to the success of 
a project of this magnitude. For any project involving key components of action 
research, it was necessary to gain acceptance, involvement and the positive initiative 
of staff. The project focused on professional development and staff inservice. A 
collegial, nonthreatening model for more effective teaching and benefit to students 
using formative supervision and observation was developed and implemented over 
three years. It focused the attention of the system on the wider range of student 
educational outcomes, affective and behavioral as well as cognitive. 


Purpose 


A wealth of information on effective teaching behaviors, substantiated by practical 
research, has emerged over the last few decades. The Spirit River School Division 
Indicators Project involved identification and validation of a set of indicators that 
can be used to assess teacher effectiveness. The criteria by which to measure 
teacher effectiveness are of central importance to the concept of teacher evaluation 
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and effectiveness. Normed measures of teacher behaviors have been established, 
and student affective and cognitive outcomes as they relate to those behaviors are 
being measured. 


The question posed is: Do the 26 teaching behaviors we have identified as being 
critical to effective teaching have an impact on student outcomes in both the 
cognitive and affective domains? 


Design 


This project concentrated on isolating and verifying established criteria for the 
various policies on teacher, school and subject evaluations which were already in 
place. Ultimately, the project involved developing and validating a set of indicators 
which could be used to assess teacher effectiveness in a formative process. 


Voluntary support through acceptance and involvement was received from staff. 

More than 80% of all staff responded to a survey that asked them to indicate the 

criteria each felt were critical to improving teacher effectiveness. Eight experienced 

and respected teachers expressed a keen interest in serving on the local committee 

whose purpose was to finalize the indicators and establish the process and terms of 
reference for the project within the established guidelines. 


This committee reviewed pertinent research and literature, developed a set of 26 
criteria of effective teaching, and formulated key variables which could impact the 
effectiveness of implementation in all division schools on a voluntary basis. The 
committee reviewed the input from staff and finalized the working project 
documents. Table 1 presents the 26 indicators. 


During the first year, on a voluntary basis and in a formative process, all staff had 
the opportunity to meet and discuss the previously developed indicators of effective 
teaching or visit each other's classrooms to observe one or more of these specific, 
established areas. Data were collected on staff participation, specific objectives 
related to each type of indicator, local or out-of-school contacts, and years of 
teaching experience. Information was also gathered on geographic distance, use 

of extra time provided in staffing, staff attitudes, and extended professional 
development activities. Two survey instruments were developed and validated 
using Good's (1963) validation criteria and parents and students were subsequently 
surveyed to gain their perspectives of indicators of quality teaching. Surveys were 
distributed to all students in grades 9, 10, 11 and 12 and parents of students in 
grades 2,5, 8 and 11. The intent of these representative surveys was to gain parent 
and student input into the process and criteria, while obtaining their perception on 
validity and/or development of new criteria. A teacher survey on satisfaction and 
professional development involvement was also conducted. The data gathered at 
the conclusion of the first year of the project were collated and presented on a 
system level, thereby assuring the anonymity of participants. 


In the second year of the project, a decision was made to enhance the credibility 
of the project as a research endeavor through the establishment of a teacher 
performance baseline to which comparisons could be made at a later point in time. 
Teaching behaviors with the lowest ratings were addressed at the system level 
through a variety of professional development activities. 
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Table 1 


Indicators of Quality Instruction 


1. Sequentially Developed Planning and Preparation 


1.1 The teacher maintains/develops short, unit and long term, flexible plans with an evaluation 
process. 

1.2 The teacher adheres to Curriculum/Program of Studies documents. 

1.3. The teacher provides a safe and organized environment with resources and equipment readily 
available. 

4 The teacher provides for individual learning needs with consideration of student abilities, 

based on diagnostic information. 

1.5 The teacher provides a variety of learning activities. 

1.6 The teacher ensures that objectives are stated. 

1.7 The teacher provides for motivation of students. 

1.8 The teacher makes learning activities relevant. 

1.9 The teacher plans for interdisciplinary activities. 


2. Key Instructional Strategies 


2.1 The teacher circulates during pupil activities. 

2.2 The teacher presents clear assignments which are relevant to student experiences. 

2.3 The teacher exhibits classroom management skills. 

2.4 The teacher provides an environment whereby active learning takes place within a working 
and non-threatening atmosphere. 

2.5 The teacher provides a physical classroom environment which is stimulating, enriched, 
relevant and has displays of student work. 

2.6 The teacher exhibits a variety of questioning teckniques. 


3. Communication Skills 


1 The teacher provides clear explanation of objectives, assignments and behavioral expectations. 
2 The teacher exhibits a variety of positive reinforcement techniques. 

3 The teacher is an active listener. 
4 The teacher provides timely and relevant reviews. 

5 The teacher uses language appropriate to the devzlopmental level of students. 

6 The teacher maintains regular parental information flow. 


4. Pupil/Teacher Relationships 


4.1 Mutual respect is exhibited between teacher and students. 

4.2 Clear routines, expectations and rules are made known and followed. 
4.3 The teacher provides for positive peer interaction and support. 

4.4 The teacher uses a fair and meaningful evaluation process. 

4.5 The teacher provides prompt feedback on student assignments. 
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A need was also recognized for additional student outcome measures relevant to the 
intent of the indicator system. The Educational Quality Indicators: Inventory of 
Assessment Instruments (Alberta Education, 1990) was consulted. Sample copies 
of a number of assessment instruments were reviewed and the Classroom 
Environment Scale (Moos & Trickett, 1987) was selected for grades 5, 8, and 11 
and the Student's Perception of Ability Scale (Boersma & Chapman, 1977) was 
selected for grade 2 to measure student outcomes in the affective domain. The 
former served to supplement data gained on cognitive outcomes measured by 
provincial achievement tests and diploma exams. The data collected in the third 
year were identical to those of the second year of the project. Table 2 summarizes 
the data collected during the second and third years. 


Findings 


Student outcomes must be monitored over an extended period of time. Only then 
can comparisons be made, trends or themes identified and results substantiated. 
As a result, only conclusions relating to strategies that have been successful in 
promoting the active involvement of professional staff in this change process are 
being shared, along with a very brief reference to parental perception. 


Staff 


The active, voluntary involvement of professional staff in action research was the 
most positive aspect of our EQI project. The value teachers attached to this project 
in terms of the professional growth opportunities it provided was clearly illustrated. 
Approximately 90 percent of all teachers responding to the survey felt their 
involvement contributed to their professional growth. 


Direct staff input into the development and validation of a set of indicators that can 
be used to assess teacher effectiveness in both formative and summative processes 
provided a sense of ownership and subsequently enhanced the likelihood of 
continued participation. The number of classroom visitations made each year is 
indicative of teachers’ willingness to be active participants in this process. 


Research provides considerable evidence to support the position that teachers 
should play a significant role in setting the criteria by which their performance 

is to be judged. Teachers in this jurisdiction did set the criteria to be used in the 
formative evaluation process, and subsequently supported adoption of these same 
criteria for the summative process. Ownership is having a positive impact upon 
teachers’ willingness to accept these as legitimate criteria. 


Teachers have shown a willingness to modify teaching behaviors. Table 3 
illustrates a change in the modal rating on a considerable number of criteria. 
Similarly, the mean rating has increased significantly on a number of criteria. 
Clear description of the meaning of each criterion resulting from discussions 
with colleagues and administrators may have contributed to this positive change. 
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Type 


Staff 


Parent 


Student 
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Table 2 


Data Collection 1990 to 1992 


Indicator 


participation 


professional 
development 


teacher 
effectiveness 


perceptions 


perceptions 


attitudes 


achievement 


discipline 


Measure 


number of visitations 
made to other teachers’ 
classrooms 


number of teachers 
attending out-of-division 
PD activities 


aggregate rating on 
each of the 26 teaching 
behaviors 


survey of parents of 
students in grades 
25 Ds Oy Ld 


survey of students in 
grades 9, 10, 11, 12 


Student's Perception of 
Ability Scale (grade 2) 


Classroom Environment Scale 


(grades 5, 8, 11) 


grades 3, 6 and 9 
achievement tests 
grade 12 diploma 
examinations 


suspensions, expulsions, 


referrals to office, dropouts 


Source 


teachers 


teachers 


evaluator 


parents 


students 


students 


students 


administrators 


Spirit River 


Teachers actively participated in the review and major revision of our student and 
school evaluation policies and processes to bring them "on line" with the indicators 
we have adopted. Two documents, a school improvement handbook and a student 
evaluation handbook were the products of this project, both germane to promoting 
effective teaching. Under divisional sponsorship, a local committee of teachers 
produced a new teacher plan book which we believe will have a positive impact on 
planning and reporting procedures. 


The teacher questionnaire indicated strong support for the professional growth 
opportunities this project has provided. Typical written responses follow. 


"A good project that promotes and facilitates teacher development." 


"T think this EQI project is very valuable to all teachers. It gives a good 
opportunity to observe others in action as well as to receive input to help 
enhance our classroom environments." 


Parents 


With few exceptions the parental survey results indicated a high level of support for 
the criteria currently being used to judge teacher performance. A safe and 
productive environment, attention to individual learning needs, clarity of objectives, 
the teacher being an active listener, and the entire area of pupil-teacher relationships 
are the highest priorities of parents. These factors have the highest impact on 
student attitudes toward school and subsequently on academic performance. 


Students 


In all three years there was generally reasonable consistency in the responses of 
parents and students. Student opinion on any of the criteria was not as strong as 
that of parents. Without exception a significant proportion of students indicated 
"less important” as their rating of any particular criterion. Overall, however, the 
only significant disagreement between parents and students was regarding the flow 
of information to the home and the importance of the teacher circulating in the 
classroom. Students attached much less importance to these than did parents. 


Conclusions 


At this point in time it is premature to speculate on the impact modification of 
instructional strategies is having upon student outcomes. As a result, only 
conclusions relating to strategies that have been successful in promoting the active 
involvement of professional staff in this process can be shared. Other jurisdictions 
may see potential value for incorporating components of these strategies into plans 
of action they may be pursuing. 
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Table 3 
Teacher Performance in 1991 and 1992 


Criteria 1991 1992 
(N=82) (N=74) 
Mean Mode Mean Mode 


Sequentially Developed Planning and Preparation 


1.1 plans 4.3 4 4.3 5 
1.2 curriculum 4.3 4 4.8 5 
1.3 environment 4.3 4 4.5 4 
1.4 individual needs 4.2 4 4.1 5 
1.5 variety 4.0 4 4.2 5 
1.6 objectives 4.0 4 4.0 = 
1.7 motivation 4.0 4 4.2 4 
1.8 relevance 4.1 4 4.3 5 
1.9 interdisciplinary oR 4 bP 4 
Key Instructional Strategies 

2.1 circulation 3.8 5 4.4 5 
2.2 clarity 4.1 4 4.4 4 
2.3. management 4.2 5 4.3 5 
2.4 atmosphere 4.3 5 4.4 n) 
2.5 environment 3.9 4 4.0 S 
2.6 questioning 3.9 4 4.0 4 
Communication Skills 

3.1 explanation 4.1 4 4.1 5 
3.2 reinforcement 3.6 4 4.1 4 
3.3 listening 4.2 4 4.3 5 
3.4 review 3.9 4 4.3 5 
3.5 language 4.2 4 4.6 5 
3.6 parents 4.0 5 4.0 4 
Pupil/Teacher Relationships 

4.1 respect 4.4 5 4.3 J 
4.2 expectations 4.2 5 4.2 bs 
4.3 interaction 4.2 4 4.3 5 
4.4 evaluation 4.0 5 4.3 4 
4.5 feedback 4.1 4 4.4 5 


Rating scale: 1 - very poorly addressed; 2 - poor; 3 - fair; 4 - good; 5 - excellent. 
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With the support and involvement of staff, we believe the project is proving to have 
a very positive and professionally rewarding impact on teachers for the benefit of 
students. It is the impact on student outcomes that will continue to be the focus of 
this study. 


Implications and Recommendations 


Based on the findings of this study, the following implications were derived. 
Reference is made exclusively to teachers as they were the "key players" in this 
process. 


Staff support is crucial to the success of any intervention strategy. Without the 
voluntary support of teachers, a study of this nature and magnitude would not have 
been possible. Mandates were not a part of this process. Staffs must be given the 
opportunity for voluntary involvement in the very early stages of any project or 
study similar to this. This enhances the likelihood of acceptance and participation 
over an extended period of time. 


Teachers must have the opportunity and the channels for "feeding" into the system. 
This generates an allegiance to the system of which they are a part. Under the 
leadership of central office administration all decisions relating to this project were 
made by a local committee of teachers. This included the identification of indicators 
of effective teaching, formulation of key variables which could impact the 
effectiveness of implementation, identification of 26 teaching behaviors perceived 
critical to positive student outcomes, articulation of clear explanations of objectives, 
development and selection of data gathering instruments, and the development of an 
action plan for implementation in all division schools. 


Direction and support from the administration and the school board is critical to the 
potential for significant changes being made. The school board's commitment to 
this project, evident in the funding it provided to cover the cost of substitute 
teachers, the addition of a staffing component to all small schools, and 

the priority given the project in the division's Three Year Plan were perceived as 
support for the project and its intent. This has provided the time teachers needed 
for planning and implementation. 


Tangible benefits or results enhance the likelihood of continued commitment and 
participation. Rewards, whether intrinsic or extrinsic, serve as motivational tools. 


Project Follow-up 


The Board of Trustees of the Spirit River School Division, with the support of 
senior administration, has made a commitment to carry on with the study. Funding 
has been budgeted to cover costs of substitute teachers and the additional staffing 
component in our small schools for the forthcoming school year. 


The extensive data collection currently in place will be continued. We are now 


completing only our first measurements of the effectiveness of modifying 
instructional strategies. In order that we may appropriately assess the results and 
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establish relationships, should the results support this, additional years of research 
and documentation are required. To stop the process at this point would have a 
negative impact on the accomplishments of the last three years. 


It is also our intent to continue to inform our publics of the efforts being directed at 
making our schools a better place for students. The local media and our school 
newsletters will continue to serve as the primary disseminators of information to the 
public. Discussions, productions and presentation at special events such as 
Education Week will continue to augment this process. Our ability to demonstrate 
to our publics that education is an excellent investment in our children and our 
future will continue to be a high priority. 


Concluding Statement 


Teachers do make a difference! This EQI project has been a positive experience for 
many stakeholders in the Spirit River School Division. We believe that the project 
is already having a very strong and professionally rewarding impact on teachers for 
the benefit of our students. 


Using the same criteria for both summative and formative evaluations has 
heightened teacher awareness regarding professional expectations. The teacher's 
planbook has been well accepted in its probationary year. It will be revised, 
enhanced and updated to best meet the needs oz staff in our division. 
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High Success in Mathematics 


Through Outcome-Based Education 


Fort McMurray RCSSD No. 32 


The purpose of the Outcome-Based Education (OBE) Project was to determine 
whether OBE makes a positive difference in student achievement, responsibility, 

and attitudes in mathematics. This report tells the story of the OBE project. Teachers, 
students and parents are the real heroes of this story. The results are a testimony to 
their beliefs, efforts and dedication. The findings, based on a variety of indicators, 
show the results of the OBE process. It is evident that the OBE process has made 

a positive difference in the outcomes of student achievement, responsibility and 
attitudes. 


Introduction 


Outcome-Based Education is a comprehensive restructuring process. It is not a 
program. Several traditional educational practices are targeted for restructuring 
through the OBE process. The practices deal with what we want all students to 
demonstrate, how well we want and expect them to demonstrate it, and when and 
how we give them credit for having demonstrated success. 


The restructuring efforts in the project were organized around the four principles 
of OBE: a) designing down from where we want to end up, b) a clear focus on 
outcomes of significance, c) high expectations and standards for the success of all 
students and d) time to learn with appropriate opportunities for learning. 


Rationale 


The Fort McMurray Catholic Schools welcomed an invitation from Alberta 
Education to collaboratively determine how well our students were achieving 
essential learning outcomes. 


We were interested in collecting data that would allow us to assess the effects 

of an OBE restructuring process. The principles of OBE were well supported in 
the educational research. The OBE team wished to align its practices with the 
principles to increase student success. Our hypothesis was: If the OBE principles 
are separately supported in the research as effective practices, they should produce 
significant results in student achievement when introduced simultaneously. 
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The OBE team understood that the restructuring process necessitated a 
comprehensive transformation of some educational practices, beginning with 
our vision of student success. Our focus on improvement and data collection 
was in one subject area since we had been cautioned to "start small". The reason 
for our initial focus on mathematics was brought about by the superintendent's 
understanding of results and his vision for improved student demonstrations of 
student achievement in mathematics. 


Purposes 


Our project purposes were to implement OBE in two schools and to review the 
effects of our efforts on student achievement, responsibility and attitudes in 
mathematics. 


Design 
Data Sources 


To collect the data annually, the OBE team decided to use a variety of measures and 
data sources to provide a comprehensive picture of student success in the desired 
outcomes. The various measures and data sources used in the project are identified 
in Table 1. 


Provincial Achievement Tests — Provincial mathematics achievement tests were 
written by students in the official writing years with repeated use of the measure 

in the following years. The tests were duplicated with permission from Alberta 
Education. Teachers received the tests on the morning of administration, supervised 
the students' writing of the test, and forwarded the tests to be scored by a district 
team. The purpose in using these provincial tests was to determine, through reliable 
and valid measures, whether the OBE process showed achievement gains over time. 
Although comparisons were made with provincial cohorts, the major comparisons 
we wanted to see were the comparisons within a school over time. 


Teacher-Developed Tests — These tests were used in grades 2 to 10. They provided 
the data used in reporting learning progress to students and parents. Other sources 
of information for progress reporting included samples of student work in 
portfolios and curriculum aligned checklists. 


Local Surveys — Local stakeholder surveys were developed and included questions 
related to each of the identified outcomes. Survey statements were rated on a 5- 
point scale: 


Not at all A little Moderately Quite a lot Very much 
(1) (2) (3) (4) (5) 


Statements to which parents responded included: 
¢« Jam aware of the High Success process in my child's class. 


« My child has shown an increase this year in achievement, responsibility 
and positive attitudes 
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Table 1 


Educational Quality Indicators/Outcome-Based Education 
Data Collection 


Student Measures Sources Grades 
Outcome 
achievement Alberta Education students 3,6,9 
mathematics achievement tests 
teacher-developed tests students 3 to 10 
local surveys students 3 to 10 
teachers 2 to 10 
parents 2 to 10 
administrators 2 to 10 
responsibility local surveys students 3 to 10 
teachers 2 to 10 
parents 2 to 10 
administrators 2 to 10 
random interviews students 3 to 10 
parents 3 to 10 
teachers 2 to 10 


support staff 2 to 10 
administrators 2 to 10 


attitudes School Subjects Attitude Scales __ students 5 to 10 
local surveys students 3 to 10 

parents 2 to 10 

teachers 2 to 10 

administrators 2 to 10 

random interviews students 3 to 10 

parents 3 to 10 

teachers 2 to 10 


support staff 2 to 10 
administrators 2 to 10 


OBE Project 64 Fort McMurray Catholic 


Statements to which students responded included: 


¢ like this process of learning. 
¢ This method has helped me to get higher math scores. 


Stakeholders were invited to provide additional comments. Some responses were 
quite elaborate and provided some valuable "thick description". 


Interviews — Random interviews were held periodically with various stakeholders 
and followed a format similar to the survey. Many of these interviews were held in 
the first year. Early feedback allowed us to make appropriate and immediate 
changes in the process. 


School Subjects Attitude Scales — This instrument (Nyberg & Clarke, 1983) provided 
external quantitative data in the affective area. The purpose was to assess students’ 
attitudes toward mathematics over the three years. Students provided responses 
related to perceived difficulty, usefulness, and evaluation of mathematics. The data 
were collected annually from students in grades 5 to 10. 


The OBE Sample 


The sample included the following stakeholder groups in one elementary-junior 
high and one high school: 


Students The OBE pilot group included approximately 600 grade 3 to 
10 students in 1990-91 and approximately 700 grade 2 to 10 
students in 1991-92. 


Teachers 12-15 teachers implemented the pilot project in 
approximately 25 classes per year, depending on semester 
scheduling and teacher p.acements. 


Parents Approximately 360 parents attended out OBE awareness and 
input sessions. They provided survey and verbal responses 
regarding support or improvement for the process. 


All parents of students in the project were surveyed at the 
end of the 1991 school year, and again at the close of the 
1992 school year. Given the number of students in the 
programs, approximately 1,300 surveys were distributed in 
the second and third years of the project. 


Administrators Two principals and two vice-principals were involved in the 
project over the three years. 

Procedures 

The three-year research project consisted of the following actions: 

Year One — team meetings to research and implement the OBE process, 


development of the indicators framework, selection of measures, and 
development of local measures 
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Year Two — team meetings to discuss strategies and processes, 
implementation of the principles of OBE, field-testing, analysis of the 
data, and team presentation at a national OBE conference 


Year Three — team meetings to determine how to refine and improve the process, 
field-test review, analysis of the data, and sharing the data 


The OBE Process 


The principles of Outcome-Based Education (Spady, 1988) guided our team 
discussions and decisions throughout the three-year project. The principles 
included: 


Designing Down — Our purpose was to promote success for all students. In order 
to promote success, we needed to clearly define the broad-based, significant 
outcomes which students would need for their future. What did future trends tell 
us students would need to demonstrate success in the twenty-first century? Then 
we "designed down" to the program outcomes. The focus on responsibility, 
positive attitudes and competent demonstrations of significant mathematics 
outcomes provided a meaningful method of organizing for instruction. 


Maintaining a Focus on the Significant Outcomes — We needed to maintain a clear 
focus on the outcomes of significance. As we became wrapped up in some of the 
details of curriculum we needed to continuously consider what we were asking 
students to demonstrate. Our collaborative work kept us on task and brought us 
back to the "drawing board" many times. 


High Expectations — OBE teachers believed that their expectations for all students 
to perform at high levels had a major impact on student demonstrations of the 
outcomes. Student work needed to be done and done well to qualify for credit. 

In math, students were expected to demonstrate mastery by performing 80% or 
higher for each program outcome in order to receive credit for the skill. In some 
cases, a skill was seen as a "can do" skill, which did not receive a mark at all. 
Rather, a checklist was used and this implied, if needed, a 100% mastery level. 
Our reporting style changed in the pilot process since averaging or summarizing 

a lot of learning did not adequately describe what a student could do. Reports 
included a listing of all the mathematics concepts the student had mastered. It also 
included a listing of the concepts which a student had not demonstrated yet, or were 
"in progress”. 


Expanded Opportunities — Teachers were astutely aware that students learn in 
different ways and at different rates. However, they were cognizant that the 
opportunities provided for students to demonstrate their learning were frequently 
cut short by time or by the methods used in the initial instruction. 


Teachers began to provide more than one chance for students to learn a concept 
or skill. Ideally, students had the opportunity to participate in a variety of active 
learning experiences during the introduction of a concept. A reteaching process, 
when needed, provided another opportunity for learning. The reteaching might 
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include such practices as peer tutoring or cooperative learning, if these were not 
included in the initial learning opportunity. Adcitional opportunities to learn 
provided the support many students needed to demonstrate success. 


The issue of providing appropriate time for students to demonstrate their learning 
was magnified by the calendar system. Some students needed time beyond a 
defined semester or school year to demonstrate the outcomes. Similarly, some 
students who mastered concepts quickly needed the opportunity to continue their 
progress. 


Dealing with the issues of time and support for learning required a collaborative and 
creative vision. 


Findings 
Student Achievement 


Teacher Tests — Our findings indicated that many more students were reaching 
mastery (80% or higher) levels in mathematics outcomes than previously on 
teacher-developed assessments. Figure 1 presents the results. 


There were substantial increases in achievement at the higher levels in both Math 10 
and Math 13 in 1991 and 1992. Teachers believed that a clear focus on what was to 
be demonstrated and high expectations for all students were making a difference in 
student performance. Teachers however were concerned about the students who 
were still reticent in accepting responsibility for learning. Teachers saw students 
counting on several chances to demonstrate success and procrastinating in 
demonstrating the outcomes. Time was an obvious factor for teachers and students 
in the semestered system. Reporting times were tedious for teachers, as students 
attempted to "catch up" on demonstration of the outcomes. In 1992, 19 students 
from the Math 10 and 13 classes withdrew to attend the school's personalized 
learning centre or Math 14 classes. Teachers felt these students exhibited behavior 
problems due to their lack of prerequisite skills or understanding of the concepts. 


Percent 
_ 8 i 
80 69 «=o 74 
60 WH Math 10 
40 CI Math 13 
20 15 . 
0 
PreOBE 1990 OBE 1991 OBE 1992 


Figure 1: Percentage of Mathematics 10 and 13 Students Achieving 
Mastery (80% or Higher) or. Teacher Tests 
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The failure rates for Math 10 were 26%, 17% and 31% respectively from 1990 to 
1992. In Math 13 the failure rates went from 37% to 18% to 26%, 1990 to 1992. 
Teachers indicated that students did not drop out of school but rather made 
decisions to get the prerequisite skills through another course before registering for 
Math 10 for a second time. The personalized learning centre was an attempt to 
provide another learning opportunity for these students prior to their enrolling in 
Math 10. This action corresponded to Radwanski's (1987) suggestion that a 
proactive process through extensive special help would assist these students' 
learning and prevent their dropping out of school. 


A review of the first semester 1992-93 grade 10 math results indicated that 80% of 
the total grade 10 students who had participated in two years of OBE instruction 
(grades 8 and 9) were performing at mastery levels of 80% or higher in all math 
courses in the first term. Thirty-nine percent of the OBE students who registered 
for math courses in the first semester enrolled in Math 10. Of these, 96% were 
performing at mastery levels. Forty-six percent of the students took Math 13; 85% 
of these students were performing at mastery levels. Fourteen percent of the OBE 
students enrolled in nonOBE math courses (Math 14 and Math 16). Of these, 25% 
were performing at the mastery levels in the first term report. 


Provincial Achievement Tests — Acceptable Standard — OBE students demonstrated 
high achievement on external measures, such as the provincial mathematics 
achievement tests for grades 3, 6 and 9; 85% of students are expected to achieve 
the acceptable standard or higher on the test. In all cases, OBE results showed 
increased achievement over time. 


Grade 3 


In 1990, 98% of the OBE grade 3 students achieved the acceptable standard on the 
provincial achievement test. This performance was repeated in 1991. In 1992 all 
grade 3 students were performing at the acceptable standard. Figure 2 displays the 
results. It may be argued that the teachers were teaching to the test. Teachers 
valued the conditions of the provincial testing process and the school and district 
attempted to maintain these conditions at all mes. Teachers were dedicated to 


Percent 100 


98 
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O District 


Math Math Math Science EB Province 
1990 1991 1992 1991 


Figure2: Percentage of Grade 3 Students Achieving the Acceptable 
Standard on the Total Provincial Math and Science Tests 
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teaching the curriculum outcomes. They believed it was the dedication to the 
teaching of the outcomes, and the provision for time and opportunity to learn, that 
caused these results to occur. In fact, in 1991 grade 3 students wrote the provincial 
science test and teachers indicated that they spent time preparing for that test. They 
were keen to see the results because they felt there was a great deal of overlap 
between science and mathematics in 1991, for example, graphing and practical 
applications. They believed strongly that OBE transfers into other subject areas. 


The grade 3 teachers shared the following comments when asked for their 
perceptions regarding the results: 


"We were directed to think critically about what the children needed to do 
easily and what we could do without. We took things that would be 
beneficial for student learning and did that EARLY so the students would 
have the skills. Problem solving was taught early because it was a skill they 
had to have. We made them discuss and verbalize what should be done, e.g., 
process. They [students] made up their own problems for other kids to do! 
Subsequent new skills in regrouping were taught in the problem solving 
format. We had students drawing, using manipulatives, talking and sharing 
ideas with a partner. Students worked in groups of 4. We tried to see who 
could come up with a way to solve the problem. We stressed the 
PROCESS, not the answer, thereby reinforcing the students’ thought 
process." 


The grade 3 teachers expressed surprise at the survey results they had collected 
in their own classrooms. Students positively ranked mathematics with physical 
education and art. They also expressed surprise at the students' reaction to the 
writing of the achievement exam. "They love the achievement test!" 


The 1992 results were similar. All students performed at or above the acceptable 
standard. 


Grades 6 and 9 


In 1991, the grade 6 students performed slightly above their local peers on the 
mathematics achievement test. There was a significant variation in collaborative 
efforts and implementation between classes at this level compared to the grade 3 
level. Basically, collegial planning did not occur other than involvement in the 
district team planning sessions. The results for the two classes are combined. 
The 1992 grade 6 OBE results showed higher performances than the local and 
provincial results of 1991. Figure 3 presents the results. 


Teacher descriptions of the OBE process at the grade 6 level included: 


"My implementation of OBE changed the achievement results and attitudes, 
e.g., self-confidence, of these students!" 


"The principles transfer into all the different subjects.” 
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Figure 3: Percentage of Grade 6 and 9 Students Achieving the Acceptable 
Standard on the Total Provincial Mathematics Tests 


Some of the changes in practice included the following: 


¢ In 1989-90 students received few expanded opportunities. In OBE 
practices, students received more opportunities. 


¢ Previously, everything counted, for example, "stuff" students didn't know was 
averaged in with the "stuff" they knew and was held against them at every 
reporting period. In OBE practices, record keeping became very focused on 
outcomes. Students got credit when they mastered an outcome. One teacher 
stated, "I didn't grade as much this year as last — only once the students were 
successful." Previously, some random record keeping occurred. "The next 
space in the book was the next activity we did!" The record keeping system 
became a diagnostic tool and identified student needs and successes. 


In 1991, 60 grade 9 OBE students wrote the 1988 Alberta Education Pilot Math 
Test. Only 28% of the students performed at the acceptable provincial standard. 
Preceding and following the test, the students indicated their perceptions that this 
test was not "worth" any credit, so they did not attempt to achieve their best. In 
1992, during a discussion following the writing of the provincial test, students 
indicated that they had tried their best. Some students felt the questions had been 
"tricky". Their 1992 performance was lower than the provincial average but 
slightly higher than the local results. The 1992 grade 9 OBE performance 
showed a dramatic increase over the 1991 results. 


Teacher practices included a great deal of assistance to students after class hours 
and the choice for students to move ahead beyond the regularly defined time 
constraints. In the 1991-92 school year, four students completed the course 
outcomes prior to the end of the year and moved to the next grade level outcomes 
midyear. Other variations in inputs included additional preparation times for OBE 
planning in mathematics. A teacher aide was available for math classes. Parent 
meetings provided awareness for parents and support for the process. 
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Provincial Achievement Tests — Standard of Excellence — The percentage of 
students achieving the standard of excellence is displayed in the following figures; 
15% of students are expected to achieve the standard of excellence on the test. 

In all cases where repeated tests were administered, results over time showed 
improvement gains in the OBE classes, as compared to earlier results. 


In 1992, a greater percentage of the grade 3 OBE students reached the standard of 
excellence in mathematics than in 1990. This was lower than the 1991 results but a 
significant increase from 1990. The OBE results in science were consistent with 
the mathematics results, although the percentage of students performing at the 
standard of excellence was lower. OBE students outperformed their local and 
provincial peers in science. The differences between grade 3 OBE and provincial 
results were significant. Figure 4 presents the results. 
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Figure 4: Percentage of Grade 3 Students Achieving the Standard of 
Excellence on Total Provincial Mathematics and Science 
Achievement Tests 
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Figure5: Percentage of Grade 6 and 9 Students Achieving the Standard of 
Excellence on the Total Provincial Mathematics Achievement Tests 
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The 1992 grade 6 results showed a dramatic increase in mathematics achievement 
over the results of 1991. The 1992 grade 9 OBE students did not match the 
provincial or local performances in the standard of excellence but they increased 
their own school performance slightly over the 1991 results. Figure 5 presents 
the results for grade 6 and 9 students. 


Stakeholder Perceptions Regarding Outcomes 


Parent and student surveys showed perceived gains in student achievement, 
responsibility and attitudes as a result of the OBE process. Although the results 
were positive, the number of parents responding was low (N=98, 15% response 
rate in 1991; N=150, 19% response rate in 1992,). Teacher responses indicated a 
perceived decrease in positive changes although interviews with teachers indicated 
that expectations for the demonstrations of the outcomes were higher in 1992. 
Figures 6 through 8 present the results. 


Teacher perceptions suggest that we may have experienced the Hawthorne effect. 
In 1991, all teachers indicated a substantial change in student achievement, 
responsibility and attitudes. Interviews with teachers in 1992 indicated their 
continued high expectations for student achievement. However, teachers believed 
achievement could be higher, students attitudes could be more positive and 
students could show greater responsibility for their own learning. In each area, 
achievement, responsibility and attitudes, teacher perceptions moved from 100% 
in 1991 to 80% in 1992. 


The most significant increase in stakeholder perceptions appeared in the area of 
student responsibility as perceived by parents. One parent indicated in 1991 that 
her child was even choosing to clean up her room now. The mother attributed this 
phenomenon to OBE, believing it had made her child feel better about herself, 
thereby increasing her desire to demonstrate responsibility at home. Student 
responses related to perceived increases in responsibility were also higher in 1991. 
Teachers felt students could demonstrate greater responsibility for learning. 


Parent responses regarding perceived increases in positive student attitudes were 
higher in 1992 than in 1991. A slight increase in student perceptions was noted. 
Teachers felt students could be more positive. 


Whether students feel better about themselves as a result of achievement or whether 
they need to feel good about themselves in order to achieve well has long been 
debated. The results appear to suggest a relationship between achievement and 
gains in the affective and behavioral domains. 
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Figure 6: Percentage of Stakeholders indicating Positive Changes 
in Student Achievement in Mathematics 
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Figure 8: Percentage of Stakeholders Indicating Positive Changes 
in Student Attitudes 
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The School Subjects Attitude Scales (Nyberg & Clarke, 1983) were administered 
annually over the three years. Table 2 presents the results. A review of the results 
led to a suggestion that since the means did not vary significantly, only minimally if 
at all, interpretations should be made with caution. Given these cautions, we make 
the following observations. There appeared to be some positive gains in the grades 
5 and 6 OBE classes while district results have stayed virtually the same over the 
three years. Grade 7 to 9 OBE students appeared to think math is less useful than 
previously. OBE Math 10 results suggest that students think math is less useful 
than previously. OBE Math 13 results suggest that some students believe math is 
more useful and they appear to like it slightly more than previously. Grade 7 to 10 
students appear to think math is not an easy subject. 


Table 2 


Mean Scores for Attitudes Toward Mathematics on 
the School Subjects Attitude Scales 


Mathematics School District PreOBE OBE OBE 


Attitudes 1990 1991 1992 1990 1991 1992 
Grades 5-6 (N=366) (N=383) (N=432) (N=84) (N=97) (N=102) 
Evaluation 27.5 26.6 27.6 21.3 27.6 28.5 
Usefulness 33.9 33.8 33.8 32.3 33.6 35.0 
Difficulty 26.3 25.3 26.3 24.6 26.8 27.1 
Grades 7-9 (N=434) (N=522) (N=461) (N=183) (N=243) (N=229) 
Evaluation 23.9 22:7 29.5 24.4 24.1 24.6 
Usefulness 32.9 32.3 34.1 34.4 33.1 31.8 
Difficulty 22.0 23.1 22.9 21.4 21.4 21.9 
Math 10 (N=102) (N=65) (N=80) 
Evaluation 23.9 20.2 21.4 
Usefulness 34.5 31.5 26.9 
Difficulty 19.7 20.5 18.1 
Math 13 (N=37) (N=44) (N=27) 
Evaluation 21.7 25.0 23.1 
Usefulness 28.6 33.3 28.9 
Difficulty 23.6 23.0 21.8 


Note. A mean score of 24 represents a neutral score whereas a mean above 24 represents a positive 
feeling toward a subject and one below 24 a negative attitude. 
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Parental Support for the OBE Process 


Parental support for the Outcome-Based Education process increased at the grade 2 
to 9 levels in the second year of implementation. There was a decrease of support 
at the grade 10 level. Figure 9 presents the results. Generally, parental support for 
the OBE process was very positive. A significant increase in support occurred at 
the junior high levels. A decrease was evident at the grade 10 level. At a meeting 
with parents of grade 10 students, parents indicated they were supportive of the 
philosophy and principles of OBE but they believed that our "time to learn" 
principle was not appropriately adhered to through the semester system. This 
constraint had also been noted by teachers. It was hoped that technological systems 
would support and improve time to learn practices. An integrated learning system 
is currently under district review. 


Table 3 presents stakeholders’ perceptions about the strengths and weaknesses of 
the OBE approach. 


Percent 
* 90 WM Very Positive 
80 

70 (1 Moderate 


Ed Not Supportive 


H No Response 


Grades 2 to 6 Grades 7 to 9 Grade 10 


Figure 9: Percentage of Parents Indicating Support for the OBE Process 


Conclusions 


There is strong quantitative and qualitative evidence that the OBE process produces 
positive results in student achievement, responsibility and attitudes in mathematics. 


The improved outcomes appear to have been brought about by teachers focusing on 
the principles of OBE. The principles include designing down from the significant 
outcomes, maintaining a clear focus on the outcomes, expanded time and 
opportunity for learning, and high expectations for the success of all students. 

Our results indicate that: 


¢ More students are achieving at higher levels of performance on teacher 
developed and external measures. 


¢ There was an increase in the percentage of parents showing support 
for the process at elementary and junior hign levels. 
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Table 3 


Stakeholder Perceptions Regarding Outcome-Based Education 


Stakeholder 


Parents 


Students 
Grades 
3106 


Grades 
7to9 


Grade 10 


Teachers 


Strengths 


high standards for achievement 
high expectations for students to work 
at it and perform well 


process builds self-esteem and confidence 


second chances are realistic 

promotes a solid understanding for math 
the clear reporting system 

teacher support for student success 


OBE is less confusing and easier 
second chances help to improve 
math skills 

useful to get a job 

it provides a challenge 

it's fun 


more is learned and remembered 

allows students to learn from mistakes 
students feel they actually learn the 
concepts 

helps students become more responsible 
will provide opportunity to get a good 
job and be proud of self 


the process is easier and students are 
confident about passing 

better learning occurs 

increases understanding in math even 
after tests have been written 

less tension when you write a test 
OBE Math is no longer a difficult 
subject to grasp 


instructional clarity 
collaboration 

focus on quality 
diagnostic emphasis 
student confidence 
love of learning 


Weaknesses 


students who move to nonOBE classes 
do not necessarily perform well in those 
classes even with having had time in the 
OBE process 

second chances are not realistic 
opportunity to "goof off" and not try 
hard the first time 

semester system goes too fast 


it's a bother, sports activities may be 
missed 


* teachers expect too much 


Source: Stakeholder Surveys, Fort McMurray Catholic Schools 
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the process continues even when 
students are not finished 


Students can get behind 
second chances are not realistic 
too many outcomes in a year 


students feel rushed but like the 
second chances 

it's demanding 

80% expectation is too high 


how to improve student responsibility 
and higher level thinking skills 

the need to show students how math is 
used in the day-to-day world 

how to deal with the OBE principles 
within a semestered system 
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¢ There were overall gains in parent and student perceptions regarding 
positive changes in student achievement, responsibility and attitudes. 


The OBE improvement process evolved over time through collaborative visioning 
and efforts. The team heeded the caution that change occurs over time and that 
sufficient time must be allocated to encourage long-term gains in ownership and 
results. 


Discussion 


The findings imply that the OBE process provides a successful framework for the 
delivery of a variety of effective instructional practices. The results of the OBE 
practices show improved student achievement, responsibility and attitude outcomes. 


Districts are invited to review the principles of OBE and determine what might be 
useful in their journey toward improved student outcomes. We hope that 
jurisdictions will take the best from the lessons we have learned and make 
adaptations appropriate to their local needs. 


Stakeholders might ask, "If OBE promotes student success, why aren't more 
schools using it?" As a result of the requests for information we have received, it is 
clear that stakeholders all across Canada are keenly interested in the OBE process. 
There is a trend toward defining what it is we want students to demonstrate in the 
future and what we need to do now to promote student success. 


Our experiences with OBE have created local ownership of Vision for the Nineties 
(Alberta Education, 1991). We encourage Alberta Education to continue the 
emphasis on educational improvement by focusing on outcomes. 


Alberta Education has modeled the collaborative improvement process through the 
Educational Quality Indicators initiative. We believe the province's collaborative 
focus on outcomes is a positive one. Local indicators provide a picture of the 
general well-being of education in classrooms and schools. The use of a variety of 
indicators can provide a significant and varied source of information from which 
teachers and administrators can make informed decisions regarding educational 
improvement. 


A limitation was the absence of reliable and valid mathematics achievement data 
which would have allowed us to make annual comparisons with students across 
Alberta. Our project would have benefited from such a provincial process. In 
particular, annual performance measures to assess higher level thinking skills 
would have been most helpful. 


Follow-up 
The EQI project has officially concluded but the teachers involved in the project 


see the need for a long-term improvement vision. They are committed to the 
continuation and improvement of the OBE process. 
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Suggestions for Improvement 


Teachers and parents have made the following suggestions for improving the OBE 
approach. 


Teachers: 

* greater encouragement for students to move beyond regular calendar-defined 
"grade" levels 

* continuation of the design down work 

* resource availability in each classroom to promote learning beyond designated 
grade levels 

* — greater integration of higher level thinking opportunities through instruction 
and assessments 

¢ technological support for continuous reporting 

* continued collaborative efforts in planning 


Parents: 

teacher training emphasis in motivation techniques for today's generation 
expand the program to grades 11 and 12, to other subjects and schools 
track the number of times students take to complete an outcome 

choices regarding participation in the process 

fun math and role playing, e.g., banker, farmer, stock market, etc. 

more time in class to complete outcomes 

change the standard to 90%; 70% 

eliminate deadlines for demonstrations of outcomes 

have fewer outcomes 
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Other OBE Support Strategies 


In addition to a board resolution to continue the expansion of the OBE principles in 

the district, the focus on student success is a major component of the district's new 

Strategic Plan. This plan, which has had wide representation from key stakeholder 
groups, is expected to significantly impact education in the Fort McMurray Catholic 
Schools. The following outcomes were identified as significant demonstrations of 

success required by graduates of our district: 


mastery of Program of Studies outcomes 

Catholic Christian service and witness 

problem solving and decision making 

critical and creative thinking 

learning to learn 

positive attitudes/behaviors, responsibility, adaptability 
effective communication 

cooperation 

continuing to learn 

self-esteem 
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Emphasis will be placed on defining these outcomes of significance and developing 
instructional strategies and data collection procedures for each of the outcomes. 


A collaborative review of current indicators and needs related to the outcomes is 
occurring through School Improvement Teams. These teams are comprised of 
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parents, administrators, teachers and students. Based on their assessment of 
current results and needs, the teams will collabcratively plan for improvement 
to enhance student success. 


School Management Plans will be focused on the improvement of delivery and 
results in the district's outcomes of significance. The management plans are 
developed annually by administrators and teachers at each school. 


Concluding Statement 


Outcome-based education is a school improvement and restructuring process; it is 
not a program. Collaborative teacher efforts are critical to the effectiveness of our 
improvement process. Teaching teams may implement the process in slightly 
different ways but the questions all teams should ask are: 


1. Are students learning what we believe is important to learn? 


2. What evidence do we accept that students are demonstrating success in the 
outcomes of significance? 


3. Based on our understanding of results, what can we do to improve the 
instructional conditions so students learn and progress continuously? 


The premises and principles of OBE have provided the foundation for our 
collaborative improvement process over the last three years. The indicators 
strongly imply that the outcome-based process implemented within a collaborative 
environment has produced positive changes in student achievement, responsibility 
and attitudes. 


The OBE process is a positive way to promote student success. We believe that 
many of our students will be better able to cope academically in high school because 
they have arrived from elementary school better prepared. We also believe the OBE 
process will better prepare our students for success after they leave our schools 
because they have demonstrated greater success in the outcomes of significance. 
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Student Growth: The Development of 
Enhanced Practices for Assessment, 


Evaluation, and Communication 


Edmonton School District No. 7 


This report of work in progress provides details of Edmonton Public Schools' 
activities and findings in the developmental work carried out from 1989 to 1992 
related to the Alberta Education "Educational Quality Indicators" initiative. It 
traces the dynamic shifts in developmental focus from simple identification of 
generic indicators and measures of student growth to coordinated development 
of mechanisms such as "portfolio assessment", the use of computer technology 
for communication and storage of information about student growth, and the 
establishment and validation of curriculum-based district level assessment strategies 
from grades 1 to 12. The report presents the assumptions and the developmental 
rationale on which the interpretation of the results is based. The district is 
continuing its initiatives in this domain. The planned direction of future 
developmental work is outlined. 


Introduction 


Edmonton Public Schools identified as a first requirement that a review be carried 
out of existing indicators and measures of district performance used in the ongoing 
performance monitoring processes. At that time, these included the results of: 


¢ annual staff and student district attitude surveys 

¢ biennial parent and general community attitude surveys 

¢ district grade 3, 6 and 9 achievement tests in mathematics, 
language arts, science and social studies 

¢ Alberta Education grade 3, 6, and 9 achievement tests 

¢ Alberta Education grade 12 diploma examinations 

¢ International Baccalaureate examinations 


Although these indicators and measures provided useful information about the 
district's overall performance, there were obvious deficiencies in the range and 
nature of student achievement strategies and measures. The curriculum-based 
tests were unable to differentiate among grade levels of achievement except at the 
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3, 6, 9, and 12 levels. The usefulness of this type of externally evaluated "paper 
and pencil" testing at widely dispersed grade levels was considered inadequate. 
The tests were also unable to extract from these results any measures of student 
performance related to six essential learning outcomes established as 
communication, responsible citizenship, well-being, knowledge, inquiry, and 
aesthetic appreciation. 


To achieve the result of the project that students, parents, staff, the board, and the 
general community have valid, accurate, and useful information about student 
achievement and growth, the following project requirements were established: 


¢ instruments and strategies that will facilitate assessment and annual 
communication of student achievement for individual student, school and 
district reporting requirements are developed 


¢ standards against which student achievement is evaluated relative 
to the graded curriculum are established 


* assessment, evaluation, and communication of continuous 
development of skills, attitudes, and knowledge are emphasized 


¢ strategies are designed to gather teacher judgments and evidence 
of student demonstration of the curriculum relative to the 
essential learning outcomes 


* opportunity is provided for students to demonstrate their highest 
level of achievement 


Rationale 


The assessments that count — the assessments that most strongly 
influence student learning and academic self-concept — are those 
developed and used by teachers in the classroom (Stiggins, 1988). 


1. Legal requirements established in Section 13 of the School Act: 


¢ The student's teacher is responsible for assessing and periodically 
reporting student achievement. 


2. Assumptions: 
¢ The achievement and performance of each student is measurable. 


¢ Student growth in achievement is one of the indicators for measuring 
student, school and district performance. 


¢ Teacher judgment about student achievement and performance is valid 
and reliable. 
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3. Principles: 
e Assessment, evaluation, and communication of student growth are based 
on the curriculum and are aligned with the school's philosophy and 
programming principles. 


¢ Information about methods of assessment and results of evaluation is 
available to students, parents and the community. 


¢ Student growth is assessed, evaluated, and communicated for all learning 
outcomes. 


¢ Evaluation and communication of student growth is ongoing and used to 
plan effective programming. 


¢ Student growth is demonstrated through a variety of performances 
evaluated by the teacher. 


¢ Student growth is enhanced when students participate in the assessment, 
evaluation, and communication processes. 


¢ Student growth is enhanced when assessment, evaluation, and 
communication are viewed positively by the student. 


¢ Methods of communicating student growth vary depending on audience 
and purpose. 


¢ Methods of assessment and evaluation of student growth vary depending 
on student learning patterns are developmentally appropriate. 


Design 


To achieve the planned results of the developmental project, the framework 
assumed the following major components: 


1. design and validation of instruments and strategies to allow for student 
demonstration of achievement, 


2. design and validation of strategies and criteria to allow for teacher judgment, 
data collection, and confirmation; 


3. investigation of workable applications of the "portfolio assessment” concept; 

4. development of prototype technology that would accommodate school and 
district needs for computer display, communication, and storage of information 
about student achievement and growth; and 


5. overall management of the project to ensure coordination of the four 
developmental areas. 
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Measuring Growth 


To enhance current district and provincial assessment practices and to overcome 

the deficiencies of the conventional paper and pencil "fill in the blank or circle the 
correct answer" type of assessment instrument, emphasis was placed on the concept 
of multiple activity tasks linked to the essential learning outcomes. 


Requirements for the design of an initial task for field-testing were established: 


¢ Tasks are seen to be valid and reliable by students, staff, parents, and the 
community and these publics are satisfied with the information they are supplied 
about student growth. 


¢ The tasks are based on the essential learning outcomes. 
¢ Teachers are involved in the development of criteria and the growth measure. 
¢ Criteria are developed to evaluate the grade level of achievement. 


¢ The tasks are undertaken by students currently demonstrating a wide range of 
performance and achievement. 


¢ The tasks are as typical as possible of normal classroom and real-life 
experiences and involve the students in multiple activities. 


¢ There is opportunity for student-to-student and teacher-to-student interactions. 


¢ Classroom teachers are trained to use student products in making judgments 
about student achievement. 


Portfolio Assessment 


One of the ways of enhancing current practices for assessing, evaluating, and 
communicating student achievement and growth is through the use of portfolios. 
To guide the development work on this topic, Edmonton Public Schools has 
defined a "student portfolio" as a purposeful collection of student products that 
exhibits to the student and others the student's demonstration of achievement, 
growth and performance in essential learning outcomes. This collection includes 
criteria for selection, achievement criteria, student self-reflections, and teacher notes 
describing the context within which the products were obtained. The student is 
involved in the selection of the contents. 


The relationship of the use of student portfolios to a generally enhanced school and 


district assessment system can be illustrated by the conceptual schema shown in 
Figure 1. 
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PORTFOLIO ASSESSMENT SYSTEM 


A CONCEPTUAL SCHEMA 


REPORT AT AGGREGATED LEVEL TO 
BOARD AND PUBLIC 


SCHOOL 
TASK 


ESSENTIAL LEARNING OUTCOMES 


, Responsible : 
Communication Citizenship Well-Being 
: Aesthetic 
Knowledge Inquiry Appreciation 


Figure 1: Portfolio Assessment System 
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The relationship of the portfolio technique to developmental assessment is aptly 
described by David Elkind: 


Developmental assessment involves documenting the work that a child has done 

over a period of time. Usually this is done by having a child keep a portfolio that 
includes all of his or her writing, drawing, math, explorations, and so on. In looking 
through such a portfolio, we can get a good idea of the quality of work that the child is 
capable of doing and of his or her progress over a given period (Elkind, 1989). 


The initial developmental work on the student portfolio focused on the following 
issues: 


¢ Application — the usefulness of the information 


¢ Responsibility — the role of students, teachers, parents, community, 
administrators, and trustees 


¢ Content — what is developed and how 
¢ Design —task development, validation, and administration 
¢ Evaluation — the criteria for making judgments 


¢ Management and Logistics — storage, movement, access, School Act 
requirements for student records 


¢ Implementation — staff development and training 
* Reporting — communication of student growth information to the many publics 


The development of workable strategies for teacher use of portfolio assessment in 
the day-to-day realities of the classroom environment was seen to take priority over 
any concerns about the physical format of a portfolio. Differentiation had to be 
made between the use of portfolio contents for district accountability and for 
individual student assessment. 


There has been extensive teacher involvement in the ongoing development of 
appropriate teacher strategies and in the identification of a realistic set of student 
products that could be incorporated within the portfolio. The following have been 
identified for further investigation: 


narrative, personal, and informational writing 

student self-reflection analyses 

photographs, videocassettes, and audiotapes 

letters of reference 

tables and reports of investigative student work 

models 

artwork of all kinds 

student generated computer disks 

teacher judgments, assessment criteria, task descriptions, exemplars, and 
context notes 
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Prototype Computer Technology 


In the initial project design, it was clear that there would be two major logistical 
restraints to obtaining maximum benefit from the proposed enhanced student 
assessment process: 


e 


human limitations on the part of the teacher in coping with a wide array of 
media and achievement criteria; and 


communicating student growth information to parents and aggregated school 
and student information to the community. 


The design, therefore, assumed the following objectives: 


to investigate the potential of current "pen-based" computer technology 


to develop prototype user screens for entry and presentation of student growth 
information 


to identify requirements for upgrading the wide area district network to 
accommodate the bandwidth required for central collection of student 
achievement data, including compressed image data transfer 


to develop teacher usable applications for local computer preparation of 
meaningful "progress reporting" to parents 


Findings 


To solely use a standardized achievement test is like casting a net into the sea 
—a net that is intentionally designed to let the most interesting fish get away. 
Then, to describe the ones that are caught strictly in terms of their weight is to 
radically reduce what we know about them. To further conclude that all the 
contents of the sea consist of fish like those in the net compounds the error 
further. We need to know more about those we catch. We need new nets. 
(Randall, 1992) 


It is important to emphasize that the new strategies designed, field-tested, and 
piloted in this developmental project are supplements to rather than replacements 
for, the traditional and current assessment practice. The results demonstrate the 
potential for enhancement of existing student assessment in our schools. 
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Table 1 


Comparison of Assessment Methodologies 


Old Nets 


assess student achievement after 3, 6, and 
9 years of schooling 


assess student achievement relative to 
grades 3, 6, and 9 curriculum 


assess student achievement relative to 
language arts, mathematics, science, and 
social studies 


school and district information about 
student achievement at four points in the 
student's career 


assessment is external to instruction — 
marked by someone other than the 
student's teacher 

assessment based on student responses on 
a "paper and pencil" measure 


individual student performance on four 
tests analyzed by outcome for use by 
teacher 


Measuring Growth 


New Nets 


assess student achievement after each year of 
schooling 


assess student achievement relative to all levels 
of the curriculum 


assess student achievement relative to the total 
curriculum 


school and district information about student 
growth in achievement on an annual basis 


assessment is part of instruction — determined 
by the student's teacher — subject to validation 
and audit at the district level 


assessment based on student performance of 
assigned tasks including but not limited to 
“paper and pencil” measures 


individual student performance on tasks used by 
teacher, student, and parent for diagnosis, 
planning, and celebration of success 


1. Field Test of Prototype Task — A prototype task was developed and field-tested 
with year one students (that is, students in their first year of schooling, excluding 
kindergarten) in nine elementary schools in June 1990. The multiple activity task 
was based on the general theme of "change" and required the students to participate 
in an activity related to at least two writing activities, drawing, self-reflection, and 


reading. 


Teachers made judgments on grade level of achievement based on the initial set 
of achievement criteria. The field test addressed essential learning outcomes, 
indicators, grade level of achievement, and identification of the sources from 
which teacher judgments were made. The essential learning outcomes and 


selected indicators are shown in Table 2. 
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Table 2 


Learning Outcomes and Selected Indicators 


Learning Outcome Selected Indicators 
communication reading 
writing 
responsible citizenship participating 
cooperating 
providing service 
well-being making choices 
exhibiting positive life habits 
knowledge applying 
calculating 
inquiry hypothesizing 
researching 
decision making 
aesthetic appreciation valuing 
expressing 


The results of the prototype task in June 1990 were analyzed with the objective of 
developing achievement criteria and a more refined task for administration in the 
1990-91 school year. The analysis and developmental work were carried out by 
working committees of teachers, district consultants, and curriculum specialists. 


Inservice training was provided in May 1991 to teachers of year one students 
in nine schools selected to pilot the more comprehensive measure which was 
administered in May 1991 to year one students (other years optional) in nine 
pilot schools. The student products were collected for analysis and central 
scoring in July 1991. 


In 1991-92, trials were carried out with tasks developed for students in years one 
to twelve in 40 demonstration sites was well as cohort participation of students now 
in year two in the nine pilot schools. Analysis of the results of these measures is 
continuing in 1992-93, in preparation for administration of the tasks as growth 
measures in April 1993. 
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2. Assessment Task Development and Process Chronology 
1989-90 + one task developed and field-tested in nine elementary schools 


1990-91 + prototype task developed and field-tested in nine elementary 
schools at the year one level 


¢ achievement criteria established and validated by teacher, 
consultant, and curriculum specialist teams 


1991-92 © 14 tasks developed and the 1989-91 tasks revised 


¢ 15 tasks piloted in 35 schocls representing "years in school" 
one to twelve (see Table 3) 


* each task cross-referenced to essential learning outcomes and 
relevant course of study outcomes and expectations 


* inservice training and consultant facilitation to complete 
judgments for each essential learning outcome and selected 
indicator provided to teachers 

¢ achievement information recorded by teachers 


¢ over 2,000 students participated in the administration of 
the tasks 


1992-93 © analysis of the teacher judgments against criteria provides initial 


verification of the validity of the assessment instruments and 
strategies (work in progress) 
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Table 3 


Assessment Tasks and Essential Learning Outcomes 


Number Task Name Essential Learning 
Outcomes Demonstrated* 


I Patterns All 6 outcomes 
2 Relationships All 6 outcomes 
MS) Adapting to the Environment All 6 outcomes 
4 Opposites All 6 outcomes 
5 Traffic Flow All 6 outcomes 
6 Points of View Communication 
7 Linking the Past to Today Communication 
8 Heroes and Heroines Communication 
9 Literary Traditions Communication 
10 A Greek Myth Communication 
11 Limits to Population Inquiry 
12 "Roto-copters" Inquiry 
13 Exploration of Shapes Inquiry 
14 Transportation Inquiry 
15 All Summer In A Day Inquiry 


* Essential learning outcomes: 1 communication, 2 responsible citizenship, 3 well-being, 
4 knowledge, 5 inquiry, 6 aesthetic appreciation. 
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3. Achievement Results of Assessment Tasks Piloted in 1991-92 


ACHIEVEMENT OF YEAR ONE STUDENTS (ASSESSMENT 
TASKS - 1991 PILOT) 


40% 50% 60% 


Reading 
Writing 


Participating 


Cooperating 


Providing Service 
Making Choices 
Positive Life Habits 


Applying 


Calculating 
Researching 
Making Decisions 
Valuing 


Expressing 


Exceeds Grade One (%) MB Grade One (%) CD Not Yet Grade One (%) 


Figure 2: Graph of Year One Achievement (Pilot Assessment Tasks) 


Table 4 
Achievement Data -- Year One Students (Pilot Assessment Tasks) 


Essential Learning Exceeds | Grade Not Yet | N= (valid cases 
Outcome Grade One| One Grade One out of 422) 
% % 


COMMUNICATION | Reading 7.5 
Writing — 11.7 53.5 
RESPONSIBLE Participating 8.3 68.9 
CITIZENSHIP Cooperating 5.8 
Providing 
Service 4.3 
WELL-BEING Making 
Choices 14.4 
Positive Life 
Habits 15.9 
KNOWLEDGE Applying 10.0 65.1 25.0 
Calculating 8.1 54.5 37.4 369 
INQUIRY Researching 7.8 65.0 21.2 371 
Making 
Decisions 14.3 63.5 2262 370 
AESTHETIC Valuing 9.2 952 31.6 370 
APPRECIATION Expressing 18.1 63.1 21.8 371 
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4. Correlation of Grading Judgments of Analysts and Teachers — All student work 
from the tasks was judged separately by the student's teacher and by analysts who 
had had major involvement in the establishment of scoring criteria. The levels of 
agreement on each indicator and the resulting correlations are shown in Figure 3 
and Table 5. 


i 


INDICATOR 
5 


[J Scores Agree Scores Differ by 1 Grade [] Scores Differ by >1 Grade 


Figure 3: Agreement of Analyst and Teacher Judgments 


Table 5 


Data: Correlation Between Analyst and Teacher Judgments 


| Scores Differ by | Scores Differ by Lasse 

Indicator 1 Grade >1 Grade Correlation 

ee ee ee 
Writing 0.7 
Participating 0.6 
0.5 


— 
Positive Life Habits 5 

74.8 23.6 0.6 

[Researching | 815 | 184 | OS | 07 | 

| Making Decisions | 81.1 —_—| 


Expressing 
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5. Student Growth Results 
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- Yr. Re ae ee 
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: 1991-Yr1f 
Researching: 99 5 —_—— 
ee She ee 
Making Decisions: 1991 - Yr. | | ____ 7s 
1992 - Yr.2 Deen ey eo 


1991 - Yr. 1 
1992 - Yr. 2 
1991 - Yr. 1 
1992 - Yr. 2 


Valuing: 


Expressing: 


Level of Achievement 


[] Grade <1 [J Grade [ |Grade2 [ | Grade3 


Figure 4: Percentage of Students at Given Grade Levels of Achievement 
Nine Pilot Schools, 204 Students, 1991 and 1992 
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Table 6 


Percentage of Students At Given Grade Levels of Achievement 


18.0 
54.0 


Indicator 
Reading: 1991 Yr. 1 
1992 Yr. 2 
Writing: 1991 Yr.1 
1992 Yr.2 
Participating: 1991 Yr.1 
1992 Yr. 2 
Cooperating: 1991 Yr.1 
1992 Yr. 2 
Providing Service: 1991 Yr.1 
1992 Yr. 2 
Making Choices: 1991 Yr.1 
1992 Yr. 2 
Positive Life Habits: 1991 Yr.1 
1992 Yr. 2 
Applying 1991: Yr.1 
1992 Yr. 2 
Calculating 1991: Yr.1 
1992 Yr. 2 
Researching: 1991 Yr.1 
1992 Yr. 2 
Making Decisions: 1991 Yr. 1 
1992 Yr. 2 
Valuing 1991: Yr. 1 
1992 Yr. 2 
Expressing: 1991 Yr. 1 
1992 Yr. 2 


Student Growth 


Grade <1 
34.0 
2.0 


28.0 
1.0 
27.0 
1.0 


19.0 
0.0 

16.0 
1.0 
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Grade 1 
48.0 
24.0 
42.0 
26.0 
60.0 
11.0 


64.0 
11.0 
63.0 
8.0 
62.0 
9.0 


60.0 
14.0 
62.0 
14.0 
60.0 
14.0 
66.0 
9.0 

63.0 
11.0 


16.0 
58.0 


16.0 
69.0 
13.0 
71.0 
13.0 
71.0 
15.0 
71.0 
16.0 
71.0 
15.0 
70.0 


72.0 14.0 
11.0 0 
74.0 


14.0 
73.0 
15.0 
71.0 
21.0 
68.0 


0.0 
16.0 


15.0 


0. 
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AVERAGE GRADE LEVEL OF ACHIEVEMENT 
1.0 


0.0 2.0 3.0 


Reading: 


Writing: 


Participating: 
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Figure 5: Growth in Essential Learning Outcomes, Nine Pilot Schools, 204 Students 
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Table 7 


Growth in Essential Outcomes (Nine Pilot Schools, 204 Students) 


Average Grade Level of Average Grade Level of 
Achievement of Year One Achievement of Year Two 
Indicator Students Students 


6. Development of Achievement Criteria — Achievement criteria for selected 
indicators of communication for grades 1 to 12 were developed in 1990-91 by a 
committee of ten teachers and two consultants. These indicators were revised and 
refined in July 1991 for grade 1, and during 1991-92 for grade 2. Preliminary 
criteria for selected indicators of inquiry for grades 1 to 12 were developed during 
1991-92 and will be revised and refined during 1992-93. Preliminary criteria for 
the selected indicators of the other four essential learning outcomes will be 
developed during 1992-93. 


7. Curricular Relevance — Developmental work has proceeded on the topic of 
relating essential learning outcomes and selected indicators to language arts, 
mathematics, science, social studies, health, and art. 


8. Assessment Task Administration 199]-92 — To maintain reliability of procedures 
and results, teachers at demonstration sites were provided with specific task 
administration routines for each of the 15 tasks. These guidelines included a 
statement of purpose, a process model, and an outline of steps and activities. 


9. District Implications — Multiple strategies and measures for student assessment 
are now included in district practices in Edmonton Public Schools. Current and 
proposed practices in each of the 193 schools are guided by the principles for 
assessment, evaluation, and communication of student growth (see Rationale). 
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Portfolio Design 


Initial work on establishing suitable physical designs for a student portfolio was an 
important catalyst for the examination of more vital issues such as the possible 
range of contents in a portfolio, the implications of regulations under the School Act 
that define contents of a student record and the access to that record, and logistical 
problems that could be associated with full utilization of a portfolio assessment 
system. Models of suitable portfolio "containers" were produced but it soon 
became clear that the most suitable physical arrangements for managing most of the 
potential contents of a student's portfolio were those developed at the local school 
and classroom level. The number of methods used is likely to be as numerous as 
the number of teachers employing portfolio assessment procedures. 


Prototype Computer Technology 


1. Pen-based Systems — The two competing systems on the market, Microsoft 
Windows for Pen Computing and Go Corporation's PenPoint, were evaluated for 
possible application as data collection systems for teacher use in the classroom. 
The findings were: 


¢ both systems were unstable 

¢ hand-printed character recognition was inconsistent and unreliable 

* the devices were bulky and unsuitable for teacher use in the classroom setting 
* acquisition costs are high 


Clearly, the pen-based systems are still using an immature technology. No further 
evaluation or development of suitable application software is contemplated until the 
second generation of pen-based computing devices is developed. 


2. Prototype User Screens — A series of prototype screens has been developed 
using the standard "Windows" screen format and Visual Basic as the development 
language. Development is continuing in parallel with the development of 
assessment tasks and related reporting procedures in the design project. Static 
screen capture images of the various interfaces developed to date are shown in 
Figures 6 to 10. 


The flexibility of the design tool is such that suitable screens and the underlying 
application for the capture and presentation of student growth information can now 
be developed very quickly, including the incorporation of scanned image and 
recorded sound when appropriate. 
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Figure 7: A Screen to Display Graphical Comparisons of Student, 
Class, School, District, or Date Level 


Student Growth 98 Edmonton Public 


CQ! TLCHNICAL PROTOTYPL: GROWTH BY SCHOOL YEAR 
DEMOGRAPHIC GROWTH CRITERIA PROFILES |PRINT] |HELP]| [QUIT] 


3 YEAR HISTORY STARING FROM: 


Student 


School 


Legend 
1990 


1991 


Communicaton Knowledge Inquiry Citizenship Well-Being Aesthetic 
1992 Appreciation 


ELBE TECHNICAL PROTOTYPE: GROWTH BY YEAR IN SCHOOL 
DEMOGRAPHIC GROWTH CRITERIA PROFILES [Print] |Help] | Quit| 
Year In 3 


< 


CHNWaNnIMwS 


1¢ae 5 67 6 G Crer se 
ead 123456789 


Knowledge 


10 
9 
8 
7 
6 
5 |. 
4 
4 bs 
2! 
1 | 
0 


SC-NwWaUNvows 


— 1 


123456789 56789 123456789 
wai 


Figure 9: A Screen to Display Student Growth by Year in School 
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Figure 10: A Screen to Display Profiles by Student, Class, School, Area, or 
District (Based on Range of Achievement by Year in School) 


3. Requirements for Wide Area Network Upgrading — The technical requirements 
for an upgraded wide area network have been identified. These address the need to 
accommodate high data traffic flow, image transfer, distributed processing of data 


where appropriate, and internal school and central connections to local area networks. 


The recent developments in data communications standards and implementation of 
fibre optic systems have made the technical concept feasible. The wide area network, 
which would interconnect all schools and central operating units independent of 
computer type, will meet the data communication needs and storage demands of 

the district's monitoring functions. Issues that continue to be addressed include: 

¢ affordability 

* cooperative line charge arrangements with the carriers 

* continuing instability of local area networks 

* security of data 


¢ the role of "multimedia" in the assessment process and the technical challenge of 
handling heavy data traffic on a wide area network 
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4. Computer Assisted Student Progress Reporting — A flexible and stable 
application has been written which allows teachers to key-enter their own 
commentary for each student and produce a centrally printed progress report, 
via local area network to the school office. The report includes the student's 
picture, comments, and "marks". The system is in use in three schools and 
incorporates many design features suggested by classroom teachers. 


There is a clear connection between this developmental work and the work related 
to user screens for presentation and capture of student growth information. The 
current progress reporting system is designed for the Macintosh computer but it is 
planned that a DOS/Windows version will shortly be available for use by a wider 
range of schools. The system is compatible with wide area network standards. It 
is anticipated that the two initiatives, progress reporting and student growth user 
screens, will be merged. 


Conclusions 
Although there remain many unanswered questions and much work still has to be 
done, the findings from the first three years of the project are encouraging and give 
promise for the future. Teacher and student comments indicate that the tasks and 
results can be useful in enhancing the reporting of student achievement and growth. 
The data derived from the teacher evaluations can be used by teachers as part of 
their assessment of the individual student's achievement and growth and can be 
aggregated at the school and district level to provide useful accountability 
information. At the same time, portfolios of student work can be used to discuss 
and validate a student's individual progress during conferences among teachers, 
parents, and students. 


The feasibility of developing easy to use computer applications for the capture, 
display, and communication of students and growth has been established. 


Follow-up 
¢ Revision and design of tasks for students in years one to twelve 
¢ Preparation of teachers for task administration 
¢ Administration of revised and developed tasks in spring 1993 
¢ Refinement, development, and application of achievement criteria 
¢ Judgment of student achievement in the demonstration schools 
¢ Collection and reporting of growth information 


¢ Development of paper and computer formats for display of growth data 
at all levels 


¢ Preparation of a staff development plan 


¢ Attempt to resolve identified issues 
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Educational Quality Indicators in 


Art and Mathematics 


Calgary School District No. 19 and Calgary RCSSD No. 1 


The Calgary Educational Quality Indicators (EQI) Project was a three-year action 
research project (1989 - 1992) funded by Alberta Education to develop indicators 

of quality student performance in school art composition and mathematical problem 
solving at the Calgary Board of Education (CBE) and the Calgary Catholic Board of 
Education (CCBE). 


Purposes 


The purposes of the EQI Project were to: 


¢ identify indicators of quality work which can be applied in the assessment of 
student achievement in areas of art and mathematics 


¢ identify conditions that appear to enhance or limit the achievement of quality 


¢ develop strategies for collecting quality student work which include recorded 
interviews with teachers and students 


* examine quality work with emphasis on alternative assessment strategies to 
paper-and-pencil testing 


The project was part of a larger, province-wide initiative sponsored by Alberta 
Education to build sets or clusters of indicators to use in a broader approach to 
assessment. 


Rationale 


The Calgary project focused on assessing performance in the areas of art and 
mathematics. These subject areas were chosen because of preliminary work which 
had already been done at the CCBE in the area of art assessment and because of 
mathematics expertise at the CBE. Further, it was reasoned that the two areas could 
allow a unique exploration of the assessment of student product and process. The 
art project could examine the product of the learning, as much of art education had 
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stressed process in the past, and the math project could examine the process of 
student thinking, as most math education had emphasized the product, or answer, 
in the past. 


Design 


The project design incorporated three phases: the development, field testing and 
pilot testing of quality indicators. Committees of teachers and subject specialists 
reviewed student work, identified characteristics of quality work, considered 
alternative approaches for recording assessment information, and proposed 
activities and materials for use in classrooms. Classroom teachers tried out a 
variety of materials and approaches with their students, materials were revised 
and then were piloted with other teachers. 


The project had a full-time designated director who was located at the Calgary 
Board of Education. A steering committee, comprised of school system 
superintendents, subject area specialists, and representatives from Alberta 
Education provided coordination and leadership. A community advisory 
committee, comprised of university professors, leaders in the fields of art and 
mathematics, parent representatives, representatives of the Alberta Teachers’ 
Association, and members of the business community provided a sounding board 
to test project ideas. Finally, two subject area advisory committees, with specialists 
from both boards, provided "hands-on" decision making for the action research in 
each area. 


Tables 1 and 2 summarize the involvement of staff, the tasks performed, and the 
results of the three years of the project. 


Findings 
Art 


The art component of the project culminated in the development of the Educational 
Quality Indicators Handbook. The handbook includes color reproductions of 
quality samples of students’ art from grades 2 and 5, and from junior and senior 
high schools. Teacher expectations for each lesson and student comments 
assessing their own work are also included. Three quality indicators emerged from 
the project as observable characteristics to be found in students' work. They were 
relationships, handling and meaning. This work is linked to the Alberta Education 
program of studies in art. Student and teacher journals were used in the pilot test to 
assist in reflective practice on the part of both teachers and students. 


Project findings for the art component included the following: 
¢ Quality assessment practice supports improved art making. 


¢ Attending to written and/or spoken student responses, reflections and thoughts 
illuminates the art work. 


¢ Quality art assessment is a shared experience between students and teachers. 
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Table 1: EQI-Art Project Outline 


Phase 1: 989-1990 Phase 2: 99()- 199 Phase 3: 
DEVELOPMENT FIELD TESTING PILOT TESTING 


Personnel || school system art specialists (4) school system art specialists (4) school system mathematics specialists (5) 
teacher researchers (7) teacher researchers (7) pilot teachers (8) 


categorized student art "above average," removed categories of student art prepared draft handbook to include all 
"average" and "below average" art assessment to focus on "what is of grades 
quality in this work" 
developed formats for assessing art: inserviced pilot teachers 
checklists, questions and narrative critique | described 3 quality indicators in narrative 
format: added teacher and student journals 
e relationships 
° handling used handbook in classrooms for 4 
* meaning months 


field tested handbooks, collected specialists visited all pilot teachers 2 or 
recommendations more times 


Results prepared 4 prototype "Quality Indicators | 4 handbooks: for grades 2, 5 junior and __| use of journals a key element for most 
Handbooks" senior high pilot situations 
2 or 3 sample lessons for grades 2, 5, 
8 and 11 recommended a SINGLE handbook — all | model of assessment practice well 
lesson included teacher expectations, | grades included received by teachers, students, parents 
classroom conditions and others 
student voice/intention added beside __| inservice and additional suggestions for 
artwork use of handbok requested expansion of single handbook: extensive 
introduction, more ideas for use of 
developed criteria for 3 characteristics: more examination of "thoughtfulness" journals, reflective practices 
¢ relationships of elements and encouraged handbook edited and printed 
principles of design 
¢ handling of the media 
* meaning or intention 


identified 5 categories of art: 

¢ directed observations 

* narrative works 

* responses to works of art 

* — special communications of the self 
¢ design problems 
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Table 2: EQI-Math Project Outline 


0 @ | 
_* DEVELOPMENT 


ase 990)- 
"FIELD TESTING _ 


Phase 3: 
___PILOT TESTING TESTING 


Personnel || school system mathematics specialists (5) | school system mathematics specialists (5) | school system mathematics specialists (5) 
teacher researchers (14) teacher researchers (14) pilot teachers (8) 


Results 


problem solving dispositions identified 
through anecdotal records of problem 
solvers 


problem bank developed 


problem format and student attitude 
questions chosen 


holistic scoring studied 


10 dispositions for all grades: 
confidence, perseverance, risk taking, 
understanding, achievement orientation, 
motivation, communication, creativity, 
organization, and reflection 


holistic scoring: 
"A" inadequate 
"B" adequate 
"C" competent 


10 problem solving dispositions tracked 


open-ended anecdotal records kept 


8 problems administered: choice of time for 


response and group work 
2 scorings for each problem 


student attitude responses collected — 
different for all grades 


classroom environment described by 
teachers 


10 dispositions too unwieldy: analysis 
beped develop 5: 

motivated 

creative 

confident 

strategic approach 

strategic process 


problems reworded 


problem formats reviewed: rewording/ 
removal of grids for answer formats 


new holistic scoring developed: 


5 problem solving dispositions tracked 
structured anecdotal records kept 


6 problems administered: choice of time 
for response and group work 


2 scorings for each problem 


student attitude responses collected — 
same for all grades 


classroom environment described by 
specialists 

dispositions gave important, useful 
information 


problem scoring positive, effective, quite 
consistent 


anecdotal records still problematic 
student attitude responses not useful 


environmental scan questioned 


¢ Effective art assessment depends on common vocabulary shared by teachers 
and students. 


¢ The collection of quality art exemplars emphasizes diversity as an attribute of a 
quality art program. 
¢ Expert professional judgment benefits from collegial work. 


¢ Quality assessment takes time. 


¢ EQI handbook exemplars are a way to share expertise and to develop 
connoisseurship. 


Mathematics 


The math component explored the concept of holistic scoring and developed some 
response criteria. These included preliminary response, partial response, complete 
response and elegant response. A series of stadents' mathematical dispositions 
were identified, including motivated, creative, confident, strategic approach and 
strategic process. Conditions for the math learning environment were also 
considered. 


Project findings for the mathematics component included the following: 


¢ Teachers report positive changes to their teaching and assessment practices as a 
result of participating in EQI-Math. 


¢ Teachers want to meet to discuss teaching and assessment. The experience of 
working closely with colleagues was very positively received. 


¢ Useful materials and practices developed in the project include student 
dispositions, holistic scoring and the exploration of diverse student responses 
and quality solutions to mathematical problems. 


¢ The use of anecdotal records, student attitude reports and generalizations about 
classroom conditions are not seen as practical, due to time and energy 
constraints. 


e Students seem to be more positive about problem solving, take more risks, and 
demonstrate more confidence as a result of the EQI project. 


¢ Systematic methods for observing students closely help teachers with teaching 
decisions and with assessment and evaluation tasks. 


e Students respond to invitations to express themselves more fully and to think 
about their thinking. 
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Common Findings 


As the EQI project evolved over the course of three years, commonalities emerged 
about the making of art and the solving of mathematical problems. They include: 


Complexity of thinking processes are common to both art and math problem 
solving. 


Art and math present assessment challenges to teachers. 


Quality assessment practice in art and mathematics supports teacher decision 
making about teaching and assessment. 


Attending to written and spoken student responses, reflections and thoughts 
illuminates the art work or the problem solution. 


Quality assessment is a shared experience between teachers and students which 
is encouraged by shared exemplars and a shared vocabulary. 


Qualitative assessment takes time for both teachers and students. 
Exemplars are a way to share expertise, develop connoisseurship, establish 


common expectations for student outcomes and celebrate diversity of student 
response. 


Other Observations and Conclusions 


Learning theory provides us with the view of the learner as an active meaning 
maker, constantly making decisions and adapting new experiences in the making 
of new knowledge. Corresponding to this view of the learner, the EQI project 
assessment procedures assisted in illuminating the learner outcomes, celebrated and 
encouraged diversity of outcomes, and helped both teachers and learners share in 
the assessment process. As a result of this sharing, researchers observed the 
following: 


Many educators are attempting to implement new assessment techniques 
through the use of student portfolios, performances and extended projects. The 
materials from EQI Calgary provide some foundational understandings and 
methods that would give these new forms of assessment increased validity. 


Informed professional judgment develops when teachers have opportunities 
to work with colleagues and share new strategies and insights. 


The EQI-Math teacher researchers and pilot teachers report that discussing 
student responses with other teachers is an important benefit of participation 
in this project. 


A logical extension of the EQI practice of assessment is its use in formal 
evaluation and reporting practices. 
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The findings and conclusions from this project suggest that the processes used 
for developing new approaches to assessment in art and mathematics may be 
valuable for other subject areas as well. 


Recommendations 


At the conclusion of the Calgary EQI Project participants at all levels assisted in the 
preparation of recommendations for future development of the project ideas and 
materials. The participants recommended that: 


the findings from this study be incorporated into evaluation and reporting 
practices 


the EQI-Art Handbook and the Problem Solving Profiles be published and 
made available to schools within the CBE, the CCBE, and to other school 
systems on request 

teacher education include a variety of assessment skills in course work 


teachers be encouraged to develop professional collections of quality student 
exemplars in art and in mathematics 


teachers be provided with opportunities te work on collegial assessment of 
student work 


school jurisdictions be invited to expand the EQI project process to include 
other disciplines 


In conclusion, this project reaffirms the importance of thoughtful reflection on the 
part of the students and teachers in the assessment of student work. It is also clear 
that innovation requires system support, guidance and time. When teachers and 
students are informed by experts, connoisseurs and the literature, they have the 
knowledge and skill to engage in meaningful assessment. 
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Academic Outcomes and Behavior 


— Identifying Quality Indicators 


Brooks School District No. 2092 


The purpose of the Brooks EQI Project was to study student behavior and academic 
achievement. We identified a set of quality indicators of responsible student 
behavior in the schools. The community was involved in promoting mutually 
supportive efforts in maintaining and improving student behavior. As an outcome 
of our project, we developed a set of measures and a set of community beliefs 
regarding the behavior of our youth. 


Context 


The Brooks School District is located in the town of Brooks which has a population 
of 9,500. The town is surrounded by 257,000 acres of irrigated farmland and over 
a million acres of rangeland used for livestock grazing. The lands surrounding 
Brooks support an active oil industry. 


The Brooks School District's six schools are located within an eight-block radius in 
the town of Brooks in southeastern Alberta. The community is stable and there is 
much pride in the school district. In September 1991, there were 2,372 students, 
127 teachers and 46 support staff. 


Purposes 


The purposes of the Educational Quality Indicators (EQI) Project in the Brooks 
School District were: 


1. to study the validity of the premise: quality education in Brooks is perceived to 
be the result of the “right” blend of attention to academic success and attention 
to development of “character” within the framework of a firm, but supportive set 
of behavioral expectations and policies; and 


2. as an outcome of addressing the premise we expect to define a valid set of 
quality indicators. 
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The expected long-term results are: 


¢ higher satisfaction and confidence among students, graduates, and employees 
with respect to the educational product as measured by staff questionnaires, 
administrator interviews, student homework completion rate, vandalism rate, 
discipline rate, former student survey, and telephone interviews; 


¢ ahigher than normal percentage of counseling/discipline related cases (for 
example, our denial of student privileges, suspension and expulsion rates may 
be higher due to our firm stand on acceptable behavior); 


¢ a lower dropout rate; 


* maintenance of high academic achievement test results, diploma exam results, 
participation rates and Rutherford Scholarships; 


¢ wider acceptance of the quality indicators developed by stakeholders in Brooks. 


The literature suggests that the result of the “right” blend of attention to academic 
success and development of “character” depends on the student and circumstance. 
The values taught at school as well as in the home are important. It is clear that 
both ingredients must exist in order for quality education to occur. The Brooks EQI 
Project concentrated on desirable student behavior and outcomes on a school district 
basis. 


Design 


The Brooks EQI Project spanned three years. The first stage of the project, 1989- 
1990, identified a set of quality indicators. Interviews were conducted with 
administrators and trustees, and teachers provided their perceptions in writing; 
these formed the basis for a questionnaire that was completed by the school district 
personnel. Information from these sources was used to identify the quality 
indicators. 


The second stage of the project, 1990-1991, involved collecting data in the following 
areas: a survey by former students; information on counseling/ discipline; homework 
completion rates; interviews on a very small sample of former students who had been 
expelled or received a Rutherford Scholarship. Data were subsequently collected on 
school vandalism. Achievement test results were analyzed for students in grades 3, 6, 
9 and diploma exam results for grade 12. An action plan was devised with 
administrators to address the information collected. 


The third stage of the project, 1991-1992, implemented the action plan. The 


community was involved in completing a profile, and developing a set of beliefs 
and a tentative action plan to follow up on perceived needs. 
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Responsible Student Behavior 


One indicator of a good school district is high expectations of student behavior. 

The Brooks School District chose to study outcomes of student behavior in terms of 
homework completion, and lack of vandalism. We also collected student opinion on 
standards of conduct in our schools. Administrators who actively counsel students 
and a supportive school board collaborate to promote positive student behavior. 


Indicators of responsible student behavior included high expectations, mutual 
respect and recognition, pride in self, others, home, school and community, 
courteous, supportive caring and sharing attributes, and mutual trust and open 
communication. 


The following model illustrates the indicators of responsible student behavior in the 
community. 


High 


Mutual Respect 
Expectations d 


and— 
Recognition 


Responsible 
Student 
Behavior 


Pride in Self, 
Others, Home, 
School & 
Community 


Mutual Trust 
and Open 
Communication 
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Figure 1: EQI Model of Indicators of Responsible Student Behavior 


¢ High expectations — The staff and trustees of the Brooks School District believe 
that adults must have high expectations of student behavior. Students must 
internalize the expectations and in turn demonstrate self-discipline. In order to 
promote self-discipline, adults must consistently help children to see themselves 
acting responsibly rather than constantly having to enforce positive behaviors. 
All Brooks staff and trustees demonstrate a commitment to developing 
responsible behavior. 
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¢ Mutual respect and recognition — Encouragement to celebrate positive and 
responsible behavior is provided through counseling and coaching. Respect for 
the rights of others and a sense of tolerance and understanding are promoted. 
Adults, especially teachers, model mutual respect and recognition. 


¢ Pride in self, others, home, school and community — The personal appearance of 
staff and students as well as the physical appearance of school buildings show a 
sense of pride. The teachers actively teach pride through, for example, the care of 
desks. Pride is encouraged through the celebration of student accomplishments, 
such as honor roll recognition, scholarships, and so forth. , 


¢ Courteous, supportive, caring and sharing attributes — Visitors comment on the 
friendliness and courtesy shown to them by students. Cooperative learning 
encourages responsible student behavior in the classroom. Integration of 
special needs students is encouraged. 


¢ Mutual trust and open communication — Students take appropriate risks and 
speak their mind, joke with adults, question policies, and so forth. Schools 
trust students to live up to expectations. Trust and open communication are 
modeled by teachers and administrators. There is a positive climate within 
the Brooks School District. 


Findings 


In the first year of the project, data collection included staff perceptions, a district 
questionnaire and interviews with administrators. This resulted in a model of 
responsible student behavior. During the second year, data collection expanded 
to include the following areas: homework completion, vandalism, counseling/ 
discipline, former student survey, telephone interviews, achievement test and 
diploma exam results, participation rates and Rutherford Scholarships. They 
measure the indicators of responsible student behavior and high expectations. 
The results follow. 


1. Homework Completion — In the fall of 1990, students from grades 4 to 12 were 
surveyed to determine the amount of homework completed in the Brooks 
School District on a Homework Completion Rate form. The data were collected 
by teachers over a two-week period. Homework was defined as “any student 
work (other than studying or reading) that requires time to complete at home”. 
The rate of return by the elementary teachers was 100%, the overall rate of 
return was 70.6%. Teachers of subject areas without homework such as band 
and industrial arts were not involved in the task. Homework completion rates 
for the five participating schools varied from 78.4% to 91.1%. The overall 
Brooks School District homework completion rate was 84.9%. 


2. Vandalism — Vandalism rates were collected from September 1990 to December 
1990. Vandalism was defined as any action which results in maintenance being 
involved in cleanup/repair, for example, broken window, graffiti, etc. There 
was no vandalism recorded during that four-month period. 
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3. Counseling/Discipline — In the fall of 1990, administrators collected information | 
regarding counseling/discipline in the Brooks School District. The objective 
was to attempt to determine the number of students disciplined by an 
administrator, and the effect of the discipline action. Discipline was defined as 
any administrator contact with students at teacher request. The schools each 
chose a two-week period in which to collect the nature of the problem, action 
taken, time involved, and if possible, results of the action. 


In total 103 students were dealt with during the two-week period: 36 cases 
related to attendance, 14 to safety, 13 work habits, 18 attitude, 14 fighting, 3 
profanity, 4 smoking and 1 drinking related problem. Action was taken in 100 
cases: 28 follow-up (included detention, student contract, loss of privileges, 
work packages and in-school suspension), 47 parent contacts, 5 agency 
contacts, 18 school district contacts (including suspensions) and 2 school board 
contacts (including expulsions). Thirty-three cases were elementary and 64 
were secondary students (6 were unidentified as they were from a school 
having both elementary and secondary students). 


4. Former Student Survey — In 1991, former students were sent a 40-item questionnaire 
asking their opinions about their schooling in the Brooks School District. The 
questions were organized into 12 categories: student behavior, discipline/ 
attendance, homework, school cleanliness, satisfaction, student problems, 
extracurricular activities, teacher/administrator, fairness/recognition, individual 
attention, parent involvement, and school involvement. Using a 4-point scale, 
respondents were asked to indicate agreement or disagreement with the statements. 
The Former Student Survey was sent to 749 former students; 175 (23%) responded. 
Almost half (49%) of the respondents had spent their entire educational career in 
Brooks, while 25% had spent high school only. The students who indicated post- 
secondary school plans included: a majority (71%) planned to attend technical 
school, college or university, 23% planned to work, 2% planned to attend another 
high school and 1% planned to return to Brooks Composite High School to 
graduate. Respondents reported that 17% received an advanced diploma with 
excellence, 42% received an advanced diploma, 35% received a general diploma, 
while 5% indicated choosing not to complete high school. 


Results indicated that students agreed most that: they learned to solve their own 
problems; teachers were willing to give students individual help outside of class 
time; the school(s) maintained a satisfactory appearance; students were expected 

to behave in an acceptable manner; students knew what was expected of them in 
school; and there was general satisfaction with the school(s), their teachers and the 
education received from the Brooks School District. Students disagreed the most 
with the following: student respect for the property of others; other students’ 
ability to understand their problems; students receiving satisfactory recognition for 
responsible behavior (e.g., telling the truth, keeping notebooks and locker clean, 
playing well with others, etc.); the emphasis placed on the social development of 
students; the help to students to make the transition out of grade 12; the school's 
guidance and counseling services were satisfactory; the job of helping students 
understand their moral and ethical responsibilities; recognition for extracurricular 
accomplishments, and additional time being spent on homework and home study 
per school night. 
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5. Student Interviews — Telephone interviews were conducted with students who 
had left school at least three years ago. Students chosen for interviews were 
either Rutherford Scholarship students or students who had been expelled. The 
purpose of the interview was to determine former students’ perceptions about 
student behavior within the Brooks schools. Thirty phone calls were attempted 
to expelled students. Many of the students had changed residences, weren't 
available or didn't call back as requested. Three requests were refused and three 
interviews were completed. Thirteen calls were attempted to scholarship 
recipients. Many of these students were at university. Zero requests were 
refused and four interviews were completed. 


Of the seven interviews, two students questioned school board policies on 
attendance and required student behavior. Two students felt that the policies 
had an effect on student behavior. All the students felt that policies had some 
effect on all students. Many felt that the home had a stronger influence than 
school district policies. Students indicated that parents supported the school 
policies in all Rutherford student cases, and one expelled student case. Five 
of the students were planning on attending college or university and two 
respondents were working. 


6. Achievement — Provincial achievement test results in grades 3, 6 and 9 and 
diploma exam results in grade 12 for 1989-90 were reviewed. Tables 1 through 
3 present the results. 


Table 1 


Percentage of Grade 3, 6 and 9 Students Achieving Standards 
on the Total Provincial Achievement Tests, June 1990 


Achievement Acceptable Standard* Standard of Excellence 
Tests District Province District Province 
Grade 3 Mathematics 97.7 91.8 34.9 26.9 
Grade 6 Science 85.6 82.5 24.8 24.6 
Grade 9 Language Arts 82.9 82.9 8.5 10.8 


* — Includes standard of excellence. 
Source: Alberta Education 
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Table 2 


Percentage of Grade 12 Students Achieving Standards on the 
Diploma Exams (Final Blended Mark), January 1990 


Diploma Acceptable Standard* Standard of Excellence 
Exams District Province District Province 
English 30 95.7 95.3 8.7 13.2 
English 33 100.0 94.0 0.0 3.8 
Social Studies 30 88.6 90.3 9.1 14.1 
Mathematics 30 83.7 86.8 25.0 22.3 
Biology 30 97.7 88.7 23.5 18.3 
Chemistry 30 94.2 89.1 21.2 20.2 
Physics 30 78.6 88.8 21.4 22.6 


* — Includes standard of excellence. 
Source: Alberta Education 


Table 3 


Percentage of Grade 12 Students Achieving Standards on the 
Diploma Exams (Final Blended Mark), June 1990 


Diploma Acceptable Standard* Standard of Excellence 
Exams District Province District Province 
English 30 94.3 94.9 5.7 12.8 
English 33 90.0 90.8 0.0 263 
Social Studies 30 96.1 89.1 11.8 15.4 
Mathematics 30 82.9 82.8 14.6 18.0 
Biology 30 96.2 84.9 24.5 19.7 
Chemistry 30 85.7 86.1 19.0 213 
Physics 30 93.8 90.7 25.0 27.6 


* Includes standard of excellence. 
Source: Alberta Education 
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7. Participation — Table 4 presents the percentage of students who wrote provincial 
achievement tests and diploma exams. The grade 3, 6, and 9 participation rates 
are from documentation provided by Alberta Education. The participation rate 
for the diploma exams was based on the number of students who wrote the tests 
compared to the September 30, 1989 enrollment. Brooks' participation in these 
tests was higher than overall provincial participation in all subjects tested except 
grade 9 language arts. 


8. Rutherford Scholarship — The total number of 1989-90 grade 12 students 


receiving a portion or total scholarship was 27; 15 student received the total 
scholarship for all three years. 


Table 4 


Participation in the 1990 Achievement Tests and Diploma Exams 


Grade Test District Province 
(%) (%) 
Grade 3 Mathematics 94.0 88.3 
Grade 6 Science 94.4 86.6 
Grade 9 Language Arts 80.7 90.7 
Grade 12* English 30 64.2 2 
English 33 26.6 21.5 
Social Studies 30 61.7 48.0 
Mathematics 30 54.5 47.9 
Biology 30 62.3 47.1 
Chemistry 30 47.4 a Be 
Physics 30 28.6 19.9 


* — Grade 12 rates are based on the total students writing in January and June 1990, divided by the 
September 30, 1989 enrollment. 
Source: Alberta Education 
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Implementation 


Administrators considered what might best be done about the information collected, 
related literature and personal perceptions, and what plans could be made to improve 
the district. Important outcomes for administrators were: increased awareness of 
“what exists” in Brooks; a vision of expected student behaviors in the Brooks 
School District; ideas, issues and problems which administrators face on a 
day-to-day basis, that is, what are the obstacles or impediments to attaining the 

goal regarding student behavior; new ideas about how student discipline could be 
handled, on a school-wide discipline basis, and an action plan. 


The action plan included involving the community in establishing reasonable 
expectations of children of different ages with respect to conduct and values at 
home and at school. A series of meetings was organized in 1992 in order to collect 
information and establish standards to involve others in the identification of what 
constitutes good behavior and what steps might be taken to promote good behavior. 


The purpose for the Character in our Community meetings was to review and 
reestablish a shared vision among schools, homes and community with respect to 
developing prosocial attitudes and behaviors among our youth. Our goal was 

to jointly develop a set of beliefs for our community. 


The process for involving the community included setting up a Steering Committee 
consisting of the superintendent, three school administrators, and the quality 
indicators director. This committee clarified the purpose of the meetings and fine- 
tuned the process. It was agreed that a series of three meetings with the community 
would help assess the support and interest that exist regarding the promotion of 
prosocial skills in our youth. It was also agreed that there would be a field-test 
meeting with identified School Advisory Council participants to first validate the 
process to be used. 


The meetings with the community established a set of beliefs as well as a set of 
statements of personal commitment. These formed the basis for potential 
community use, and reaffirmed the support for school district efforts in terms of 
promoting prosocial skills. These belief statements are presented in Figure 2. 


Conclusions 


In our efforts to determine whether we have the "right" blend of attention to 
academic success and attention to the development of "character" a baseline of 
information was established for the Brooks School District. It included how the 
staff and administrators viewed the district, how much student homework was 
completed, the amount of vandalism and discipline that occurred, how the former 
students viewed the district and achievement rates. 


Staff and administrators in the Brooks School District believe that students behave 
responsibly. This is shown in the indicators which were identified: mutual respect 
and recognition, pride in self, others, home, school and community, courteous, 
supportive caring and sharing attributes, mutual trust and open communication and 
high expectations. The indicators were measured through homework completion, 
vandalism, discipline, Former Student Survey and student achievement rates. 
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CHARACTER IN OUR COMMUNITY BELIEFS 


Character is the ability to identify the elements of a circumstance or situation, to 
determine one’s most constructive role in it, to carry out that role directly, to sustain 
it as long as necessary, and to learn from the consequences of one’s actions. 


Within the context of our community which has good people of all ages, we are 
committed to attaining an even higher standard of behavior through action 
consistent with the following. We believe that: 


Ls 
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Good character is respecting yourself and others. No one has the right to take 
(without consent), harm or destroy another person's physical or mental well- 
being. No one has the right to devalue the character of another. 


. Noone has the right to take (without consent), harm or destroy another 


person's property. 

The difference between rights, responsibilities, and privileges must be recognized. 
The home must provide the foundation for the development of character. 
Rules/laws need to be fairly and consistently applied. 

Respect is a mutual endeavor especially between adult and child. 


People must have and show respect for rules and laws in the community and for 
those who have responsibility for enforcing them. 


The best way to teach character is to be a good role model. 


Positive self-development evolves from achieving a sense of purpose, from 
having a positive self-concept, and from being accepted. 


. Community contributions are made by those who wish to institute improvement. 


. Young people need and want age-appropriate limits and expectations on their 


freedom and activities. 


. An individual is accountable for personal actions and must live with the consequences. 
. Appropriate language and manners are expected in public places. 


. Restricting the use of alcohol and drugs to responsible use is necessary, desirable and 


achievable. 
Social activities of young people must be effectively monitored by a responsible adult. 


Offensive behavior will not be tolerated and those responding appropriately 
deserve the support of the community. 


Figure 2: Community Beliefs in Brooks 


Academic Outcomes and Behavior 119 Brooks School District 


The 1990 homework completion rate baseline for the district was 84.9%. There was 
no vandalism from September to December 1990. There were 103 cases of 
counseling for inappropriate student behavior and parental contact by administrators 
during a two-week collection period. 


The 1991 Former Student Survey indicated that attention was paid to academic 
success, and that there were firm behavioral expectations. However, the respondents 
felt that the teachers in the Brooks School District need to pay further attention to the 
development of character (especially in terms of promoting respect, tolerance, 
recognition and guidance). Student achievement was relatively high on provincial 
exams in 1989-90. 


Implications 


The Brooks School District personnel were concerned that students behave in order 
for learning to occur. It is timely in that throughout Alberta and western Canada 
there is an increasing number of students with behavioral difficulties. The efforts 
made by our school district in the promotion and support for positive behavioral 
outcomes could be adapted in other school districts. It is important to remember 
that student behavior is first and foremost determined by the family, and it is 
therefore important to establish a school-family-community partnership. 


One of the expected long-term outcomes of the Brooks project was higher 
satisfaction among students, graduates and employees. This will need to be 
monitored over time and compared to the original results. While a higher than 
normal percentage of discipline related cases was expected, this did not prove to be 
the case, but needs to be monitored over time. Dropout rates were not addressed 
during the project, as there was no agreement reached regarding a definition. 


Maintenance of high academic achievement test and diploma exam results, 
participation rates and Rutherford Scholarships, and wider acceptance of the quality 
indicators developed by stakeholders in Brooks were expected long-term results. 
These also need to be monitored over time now that measures have been 
established. 


Recommendations 


There is increasing awareness by students, parents and teachers of the importance 
of good behavior for students to learn. This project looked at what good student 
behavior is. We have strong support from the staff of the Brooks School District. 
The objective of the stakeholders is to positively influence student behavior through 
collaborative community efforts. 


It is recommended that other districts concerned with promoting positive student 
behavior adopt and modify measures to suit their school district. These might 
include: staff questionnaires, administrator interviews, student homework 
completion rates, vandalism rates, discipline rates, former student surveys and 
telephone interviews. It is also recommended that school districts form 
partnerships between their schools, parents and communities. It is important for 
schools to have some representation (in an unofficial capacity) with the parents in 
order to work together in an effort to promote responsible student behavior. 
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For Alberta Education it is recommended that: 


¢ the information from the Brooks Project be shared with interested school districts 
throughout the province; 


* support be made available to provide for further collaboration and follow-up 
among projects with common interests; and 


¢ further research projects be initiated by the province so that Alberta can maintain 
its place at the forefront of efforts to improve the quality of education. 


Follow-up 


The Character in our Community (CIC) formed a follow-up committee under the 
auspices of two school administrators. The purpose of this committee was to 
promote acceptable young people's behavior throughout the community. The 
School Advisory Councils (SAC) in each school have a representative on the 
committee. It will be their responsibility to communicate committee initiatives to 
their school SAC. 


The school district has a portfolio of information collected at a point in time, such as 
discipline rates, which could easily be duplicated. Serious consideration has also 
been given to surveying students after they graduate on a yearly basis to monitor the 
success rate regarding the promotion of positive student behavior over time. 


Concluding Statement 


Our original premise was that students need to behave in order to learn. We found 
that students, families and the community have the same belief. The information 
collected indicates how successful we were in terms of our combined efforts. The 
Brooks EQI Project sought to establish whether or not the "right" blend of attention 
to academic success and "character" within the framework of a firm but supportive 
set of behavioral expectations and policies was implemented. To this end, we 
reviewed the present policies as well as collected information from parents, 
students, and teachers on responsible student behavior and academic success. 

The collection process has heightened the awareness of staff and community 
regarding student behavior. The belief has been, and still is, that we have a good 
school district that promotes positive student behavior. The Brooks School District 
has identified a set of indicators to measure this belief. 


There is a strong interest in promoting a positive community attitude in developing 
responsible student behavior. The district will direct its efforts to monitoring 
positive student behavior, and will continue to heighten the awareness of the staff 
and community as it applies to positive student outcomes. It is important that any 
school district be committed to involving the community in efforts to promote 
responsible student behavior since much time and energy is involved. Increasing 
efforts will be required by educators to support the family and community to 
promote responsible student behavior. 
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A Study to Identify and Measure 


Desirable Student Social Outcomes 


Fort McMurray School District No. 2833 


This report outlines one school jurisdiction's efforts to develop a set of social 
skills, behaviors, attitude indicators and standards that can be used to assess the 
quality of a selected portion of that jurisdiction's educational program and 
delivery. It describes the development of an expanded systems model of social 
competence, the use of standardized instruments and finally the development of a 
three-level approach to measuring and reporting social skills development. 


Introduction 
Rationale 


The focus on assessing and reporting student social skills reflected the district's 
recognition of the broadening mandate of schools in modern society. Schools are 
becoming increasingly responsible for the broad education of students that now 
includes a social dimension. Despite an extensive body of research on social 
competence, and its suggested significant role in student academic, social and 
psychological development, current definitions of social competence, as well as 
strategies for its measurement and reporting in the school environment, are 
limited. In recognition of this, and as part of the Educational Quality Indicators 
initiative, the Fort McMurray School District No.2833 undertook a study to 
examine the measurement of student social competence. The major purpose of 
the three-year project was to develop a set of social skills, behaviors and attitude 
indicators and standards that could then be used to assess the quality of a selected 
portion of its educational program and delivery. 


Purposes of the Study 


In order for the study to meet the purposes outlined above, the following tasks 
were completed: 


1. identify or develop a set of desirable student social skills; 


2. identify or develop measurable outcomes and standards for the desirable 
student social skills; 


Social Outcomes 122 Fort McMurray Public 


3. establish a methodology for collecting, analyzing and interpreting the data; 

4. identify or develop strategies for teaching the identified desirable student 
social skills; 

5. identify or develop a means of reporting information and findings to users. 


A Model of Social Competence 


Social skills are a part of human behavior that are hard to define and measure. A 
model helps to conceptualize and understand the relationships among parts. The 
Fort McMurray Public's EQI Project used a systems perspective based on a 
modified CIPP model (Worthen & Sanders, 1987) as the conceptual framework. 
The systems perspective was used as the basic organizational framework for the 
project because it "was compatible with the beliefs of the study team and appeared 
to provide good information for program development and student evaluation 
related to social competence" (Young, 1990, p. 2). 


Certain commonalities are shared among the theories and models of social skills 
and social competence. 


There are specific elements and skills. 

There is a need for the elements/skills to be processed in an appropriate 

manner; the processes may be either cognitive or affective. 

The appropriateness of behaviors to specific situations is crucial. 

There is no agreed upon generic social skills listing. Also, any list needs to 

be flexible to some extent to the needs of the user. 

Appropriate social responses are affected by the developmental level of the 

respondent, and the audience for the respondent. 

Social responsibility can be learned. 

The whole social competencies construct is dynamic, synergistic, and 

organic. 

Specific social elements and skills are comprised of specific and discrete 

verbal and nonverbal behaviors and entail both effective and appropriate 

initiations and responses. 

9. The skills are interactive by nature. 

10. Deficits and excesses in social response can be specified and targeted for 
intervention. 

11. Instructors attempting to teach social skills or appropriate social responses 

should adhere to the "relevance of behavior rule" (Allyon & Azrin, 1968) 

which states that instructors should teach only those behaviors that will 

continue to be naturally reinforced after training. Adopting this viewpoint 

assists in ensuring that social skills selected for instruction will have some 

intrinsic value to the child, some benefits for the child, and be valued by 

others who would likely reward their occurrence (Young, 1990, p. 4). 


PAS wre Ee 


The systems model of social competence was a compilation of a number of 
research-based ideas about the process (Perry & Perry, 1987; Hargie & McCartan, 
1986) and the factors (Reschly & Gresham, 1981) which influence an individual's 
social interaction skills. The social outcomes deemed desirable were those 
outlined in the 1988 Alberta School Act (Section 7, p. 11) and in the 1989 Guide 
to Education. Figure 1 presents the model. 
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Social Situation (Context) 


1. Audience Factor: - conduct 
- acceptance 
- reinforcement 


2. Task-Related Factors: - academic performance 
- classroom skills 


3. Environmental Factors: - hostile/friendly 
- familiar/unfamiliar 


Social Information 


1. Personal Factors: 


- self-esteem 
- emotions 


Social Information Processing 


1. Encoding: 
- qualification 
- classification 


- appearance 
- developmental level 2. Interpreting: 
- age - relationships 
- Sex - transformations 
- physical performance - causations 
- academic performance - critical thinking 
- goals/motivation/needs 


2. Interpersonal Factors 
- basic interaction skills: 
- greetings and partings . 
- self-disclosure 
- non verbal communication 
- coping skills: 
- asking questions 
- explaining 
- assertiveness 
- interacting in groups (cooperation) 
- responding skills: 
- listening 
- empathizing 
- reflecting 
- reinforcing 


3. Generating Alternatives: 
- problem solving 
- creative thinking 


4. Choosing: 
- decision making 


Figure 1: A Systems Model of Social Competence 


1. Behaviors: 
- basic interaction skills 


- coping skills 
- responding skills 


- classroom/academic skills 


- thinking skills 


2. Perceptions: 
- goals/motivation/needs 


- appearance 
- self-esteem 


3. Social Appropriateness: 


- acceptance 
- behavior conduct 


- positive reinforcement 


Measuring Social Skills 


The conceptualization of social competence within a systems framework resulted 
in the identification of three major facets of the concept: social skills, social 
information processing and socially desirable outcomes. These initial three facets 
of the social competence model were within a situational context. 


Given the preliminary definition of social competence, and the identification of a 
broader system framework in which to approach its teaching and measurement, 
the study team addressed the specific task of developing an assessment 
methodology for measuring and reporting student social competence. Drawing 
on the findings of the literature review, which indicated the complexity and time- 
consuming nature of developing reliable and valid social skills measurement and 
reporting systems, the study team decided to examine existing instruments to 
determine their appropriateness for use in the Fort McMurray context. Potential 
instruments were reviewed that fit the following criteria: 


1. The instrument focused on students in regular classrooms. 

2. The instrument accommodated a variety of developmental levels, largely 
defined as divisions I (grades 1 to 3), II (grades 4 to 6), II (grades 7 to 9), and 
IV(grades 10 to 12). 

3. The instrument included a variety of sub-scales and items that addressed the 
social response areas identified, and had the potential to be adapted to meet 
the requirements of the team’s particular needs. 

4. The instrument focused on positive social behavior, as opposed to negative 
social behavior, or lack of social behavior. 

5. The instrument included variations pertinent to the various audiences who 
might be asked to respond such as parents, teachers and students. 


The study team identified two instruments having the potential to assess student 
social skills, particularly perceptions — the Self-Perception Profile for Children 
(1985) and the Self-Perception Profile for Adolescents (1988); both developed by 
Susan Harter. Subsequent consultations with the author confirmed the potential of 
both instruments for use as part of the research project given their capacity for 
adaptation. A third instrument, the Teacher Rating of Social Skills — Children 
(TROSS-C) developed by Clarke, Gresham and Elliott (1985), was also identified 
as an appropriate measure for observation of student behavior. The TROSS-C was 
revised by Gresham and Elliott (1990) based on further development work, and 
renamed the Social Skills Rating System (SSRS). The revisions to the instrument 
did not substantively alter its form or content. 


As a further assessment of their suitability, the selected instruments were cross- 
referenced to the desired social outcomes developed as part of the model of social 
competence (Table 1). The results of this cross-referencing exercise show the 
instruments to provide good coverage of the theoretical construct. 
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Table 1 


Cross-Referencing Social Outcomes and Assessment Instruments 


Desirable Social Outcomes ! Instruments 
1. BEHAVIORS: 
1.1 basic interaction skills 1.1 SSRS 2 (Social Initiation) 
1.2 coping skills 1.2 TROSS-C 3 (Academic 
Performance, Peer 
Reinforcement, Cooperation); 
1.3. responding skills 1.3. TROSS-C (Peer 
Reinforcement, Academic 
Performance) 
1.4 classroom/academic skills 1.4 TROSS-C (Academic 
Performance) 
2. PERCEPTIONS: 
2.1 goals/motivation/needs 2.1 SPP 4 (for Adolescents - Job 
Competence, Romantic Appeal, 
Friendship) 
2.2 appearance 2.2 SPP * (Physical Appearance) 
2.3 self-esteem 2.3 SPP (Self-Esteem) 
3. SOCIAL APPROPRIATENESS: 
3.1 acceptance 3.1 SPP (Social Acceptance) 
3.2 behavior conduct 3.2 SPP (Behavior Conduct) 
3.3 positive reinforcement 3.3 SPP (Peer Reinforcement) 
1 Appropriate desirable social outcomes at various developmental levels and as outlined in the 
Alberta School Act (1988) and the Guide to Education (1989). 
2 Social Skills Rating System (Gresham & Elliott, 1990). 
3. Teacher Rating of Social Skills — Children (Clarke, Gresham & Elliott, 1985). 
4  Self-Perception Profile for Adolescents (Harter, 1988). 
5 Self-Perception Profile for Children (Harter, 1985) and Self-Perception Profile for 
Adolescents (Harter, 1988). 
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Field Test of Standardized Instruments 


In the second phase of the project, the three identified instruments were field- 
tested in the fall of 1990. The major objectives of the field test were to confirm the 
quality of the data gathered by the instruments, and to review the ease and 
practicality of instrument administration. The appropriate SSRS instrument was 
administered to a grade 3 class and a grade 11 (semestered) class; the SPP-C to a 
grade 6 class and the SPP-A to a grade 11 (semestered) class. Prior to the field 
test, teachers were given a brief overview of the purpose of the study, as well as a 
summary of the instruments to be used. As part of the instrument administration, 
teaching staff were asked to complete one form for each student, have the student 
complete one, and have the parent complete one. One follow-up was completed 
for parents who did not return the forms. Once the completed forms were returned 
from the three groups of participants, an analysis was completed for each student 
in the sample. 


Completed assessments (teacher, student and parent instruments) were collected 
for 34 of the 84 students assessed (a completed sample of 40%). Two major 
problems were identified through the administration: training in test 
administration for teachers; and ease of instrument administration. Teachers 
reported that they felt uncomfortable both in administering the instrument, and 
in analyzing and interpreting the results. They further identified some potential 
suspicion and antagonism among parents relating to the process. Teachers 
suggested the need for the introduction of comprehensive training sessions if 
the instruments were intended to become a standard component of student 
assessments. Teachers did acknowledge, however, the value of the information 
gathered by the individual instruments. 


Teachers deemed administration difficult because of the amount of time involved 
for each teacher both in assessing individual students, as well as ensuring parent 
returns. For the field tests, parental returns were low (34 out of a possible 84) 
with five outright refusals. Teachers also reported that the administration of the 
instruments in a semestered high school in November, was too early in the school 
term for them to make informed assessments of students’ social skills. 


Based on the results of the field test of these instruments, all three were 
considered “impractical” for use with whole classes as part of a universal process 
of student social competence assessment and reporting - although their use with 
smaller groups of students was considered feasible. In their discussions of the 
field-test, teachers suggested the development of a simpler-to-administer Social 
Competence Checklist, with the use of commercial tests by counselors as a “back- 
up” to the simpler checklist, and it was to this strategy that the project team turned 
next. 


A Three-Level Approach to Measuring Socia! Skills 


From the field test of the instruments and subsequent discussions, a three-level 
approach to measurement evolved, focusing primarily on issues of practicality, 
administration and quality of information. The latter referred to the original 
instrument selection criteria. The steering committee began to develop a checklist 
to help teachers and parents more easily idenzify children's level of social skills. 
The checklist was based on the same criteria to which the standardized 
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instruments had been cross-referenced (Table 1). It attempted to measure those 
skills which would result in desirable student social outcomes. The checklist 
eventually became the Social Skills Diagnostic Screen. 


Additionally, district report cards were undergoing a detailed review. The portion 
of the report cards where teachers comment on children's social skill development 
was revised to reflect a more detailed and research-based assessment using 
selected components of the Social Skills Diagnostic Screen. 


Level One — Each teacher reported to parents, at report card time, the social 
development of their children. Each teacher rated students as being well 
developed, developing appropriately, or needing further development. The 
ratings were based on: 


* personal factors 

* interpersonal factors 

* social reasoning skills 
¢ classroom behaviors 


Level Two — If any of the students, parents, or teachers wished or required a more 
detailed assessment, the District's Social Skills Diagnostic Screen would be 
completed by the teacher for use in providing a rating for the report card, forming 
the basis for discussion with parents and/or the student, or for developing a 
moderate program of remedial action. Additionally, a form could be completed 
by the student, the parents, and a Composite Student Profile compiled to be used 
as the basis for discussion. Using this information, a program of intervention or 
skill development exercises could be initiated. 


The Social Skills Diagnostic Screen consists of four sections: personal factors, 
interpersonal factors, social reasoning skills, and classroom behaviors which 
represent more expansive investigations of the same four areas of reporting 
identified at level one. Each section contains a series of simple to administer and 
score questions. The screen may be administered in either individual or group 
settings, and may be completed by the student, a student's peer, the teacher, a 
parent or any other individual known to the subject in other than a casual manner. 
The screen is designed to be applicable to school-aged subjects, across all grade 
levels. The time taken to administer the screen varies from approximately 10 to 
15 minutes with older subjects, to 20 to 30 minutes with younger subjects who 
may need to be interviewed on an individual basis. 


Level Three — If detailed knowledge of the student's social development was 
required, the student's case would be referred to the counselor for further 
assessment and the development of an Individualized Program Plan related to 
the development of appropriate and acceptable social behavior. The battery of 
assessment instruments to be used may include: 


¢ Social Skills Rating System (Gresham & Elliott, 1990) 
¢ Self-Perception Profile for Adolescents (Harter, 1988) 
¢ Self-Perception Profile for Children (Harter, 1985) 


Counselors would be trained in the assessment instruments and the related 
procedures through inservices provided by district consultants. 
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Findings 


The major outcomes of the research were twofold: first, the conceptualization of 
the systems model of social competence that drew on existing literature relating 
to the concept and provided a framework for its definition and subsequent 
measurement; and second, the development of a three level approach to assessing 
and reporting student social competence development within the school 
environment. As part of this three-level approach, the Social Skills Diagnostic 
Screen, was developed and subsequently pilot-tested. In addition to the 
development of this instrument, an activity manual for teaching social skills 

was prepared, for use in conjunction with the screen. 


The development of the social competence model, and the associated assessment 
methodology was completed as part of a series of stages. Initially, the concept of 
social competence was defined based on the findings of a substantive literature 
review and subsequent discussions of the concept by the project study team. As part 
of the second phase, standardized social skill measurement instruments were pilot- 
tested to determine their appropriateness for use in the school environment. As a 
result of the findings of these pilot tests, a more flexible and comprehensive 
assessment model was introduced that included a three-level assessment protocol 
that moved from the general to the specific. The second level of the model included 
a specially developed assessment instrument, the Social Skills Diagnostic Screen. 


The findings of the three-year research project suggest the value of the three-level 
approach to measuring and reporting student social competence. Field-testing of 
standardized commercial instruments revealed considerable difficulty in their 
practical application in the everyday school setting — given both the unfamiliarity 
of teachers with the instruments themselves, as well as their unwieldy and 
complex nature that was reflected in the time taken to administer them. By 
contrast, the Social Skills Diagnostic Screen, developed in part from a review of 
existing instruments, proved more practical for classroom use, and in pilot testing 
demonstrated some measure of statistical validity (content) and reliability (test- 
retest). The results of the limited pilot testing of both the screen and the broader 
three-level approach to assessing social skills, suggested their value as part of an 
integrated assessment and reporting package. 


Conclusions 


The findings of the project cast considerable light on the complex nature of 
measuring and reporting student social competence. Challenges similar to those 
identified within the literature were initially encountered in defining social 
competence. The adoption of a systems perspective provided a broad framework 
for the conceptualization of social competence. It provided the framework for the 
inclusion of the context, input, process and product dimensions essential to any 
model of social competence. As a theoretical concept and practical component 
of everyday life and interactions, social competence retains an inherent dynamic 
quality that mitigates against its simple description. Rather, the concept needs to 
be expressed in ‘fluid’ terms that reflect its dynamic, and complex nature, as well 
as its link to day-to-day interactions. 
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Given the challenge in defining social competence, as well as the specific 
elements that contribute to it, the current project also revealed the complexity of 
its subsequent measurement and reporting. The initial field test of standardized 
commercial instruments failed to meet the project need for a practical assessment 
methodology that provided the basis for meaningful dialogue between teachers 
and parents on student social development. The subsequent development of a 
three-level approach represented a creative and innovative strategy that, based on 
limited field testing, proved practical, valid and reliable in the measurement and 
reporting of student social competence. While this approach performed 
satisfactorily within the pilot test, it requires further review and testing to confirm 
its validity, reliability and practicality as an assessment and reporting strategy. 


Implications 


The conceptual model of social competence developed within the current study, as 
well as the three-level approach to assessment designed to accompany it, both 
have potential for application within the school environment. Specifically, the 
three-level approach is research-based and cross-referenced to desirable student 
outcomes, and thus, may be effectively used in varied school settings for assessing 
and reporting student social skills. 


Empirically, the development of a valid, reliable and practical assessment and 
reporting methodology for student social competence is an important break- 
through. While several standardized commercial instruments are available, their 
application in the school environment is limited given their complexity in terms of 
administration and practicality for the classroom teacher. 


As an extension of the advantage gained through the development of a practical 
assessment and reporting methodology, it may be anticipated that teachers will be 
able to address instruction in social skills more effectively and confidently. Not 
only will teachers be in a position to determine the student level of social 
competence, but they will also be able to review possible causes of skill deficits, 
as well as the impact of specific interventions designed to address deficits. In the 
long term, the development of an effective assessment and reporting strategy may 
thus impact on the nature and content of social skills instruction in the classroom 
and school environments. Within this context, the Social Skills Development 
Activity Manual (Sherman, 1991) developed in conjunction with the Social Skills 
Diagnostic Screen, serves as a valuable resource for teachers and parents. 


Recommendations 


Based on the preliminary findings of the research to date, the major 
recommendation of the current study is that research continue in measuring and 
reporting social competence. Student social competence forms a key element of 
general educational development, and one that has frequently been overlooked 
given problems associated with both its definition and measurement in the school 
environment. Despite the encouraging results of the current research, more 
extensive field testing of the Social Skills Diagnostic Screen is required. Scoring 
norms need to be established for different user groups, as well as for individual 
sections of the screen so that it may be more effectively used to pinpoint 
connections and specific areas requiring further assessment. More research is 


Social Outcomes 130 Fort McMurray Public 


also required into the use of interventions to address student skill deficits. 
Specifically, the nature and extent of interventions needs to be clarified for 
individual students based on their individual needs. 


To assist in this broader testing of the three-level approach, as well as the screen 
itself, it is recommended that both be field-tested in a variety of school settings, as 
well as individual settings in long-term longitudinal studies. The findings from 
these more extensive field tests should yield valuable results regarding the 
validity, reliability and practicality of the assessment process. 


Follow-up 


Within the Fort McMurray context, follow-up to the project will be ongoing. 
Parents have requested additional information on strategies they may use with 
their own children to work on appropriate social skill development. Similarly, 
professional and support staff from across the school district have expressed the 
wish both to facilitate and take part in initiatives relating to the whole area of 
social skills development and assessment. Az the district level, the board has 
determined that student social competence is a priority for the 1992-1993 school 
year. As part of this, the three-level approach will become an integral component 
of the policy and practice of the district. 


Concluding Statement 


The model of social competence developed within the current study, and its 
extension into a three-level approach to assessing and reporting for use in the 
school environment, forms an important initiative in the ongoing review of 
educational quality. The assessment and reporting of student social competence 
within the school context has, to date, been limited, with educators forced to rely 
on standardized commercial instruments that frequently do not meet either the 
needs of students or staff. The need for students to develop an appropriate level 
of social competence is critical, given its close relationship with other areas of 
educational performance and growth. 


The three-level approach to measuring social skills, and its second-level Social 
Skills Diagnostic Screen, form one response to the need for positive indicators and 
standards that address student social skills in the school environment. This project 
has focused on the development of an assessment model that is straightforward to 
use and practical. If the research has raised an awareness of the importance of 
social skill acquisition as it relates to student development, then a further goal has 
been met. 
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Signs of Learning in the Affective Domain 


Lethbridge RCSSD No. 9 


The Lethbridge Catholic Schools Project recognizes the importance of learning 
outcomes in the affective domain and the efforts of teachers to comply with 
assessing and reporting those outcomes. The project was designed to operate 
within the classroom setting and to be subject to its limitations. This report presents 
the background, the methodology of the project, the findings and their implications. 


Introduction 


The Lethbridge Catholic Schools researched s:gns of learning in the affective 
domain as part of the Educational Quality Indicators collaborative initiative to 
develop an indicator system which will provide information about education 
to Alberta Education. 


Purposes 


The first purpose of this study was to find indicators to identify attitudes in the 
affective domain that are set out by Alberta Ecucation in the goals of schooling. 


A second purpose was to devise a system for monitoring and recording behaviors 
which are the expression of the attitudes. 


The third purpose of the study was to assist educators and parents in their efforts to 
teach and model behaviors which express positive attitudes and which assist in the 
formation of positive attitudes. The formation of positive attitudes is an important 
goal of education, and educators are expected to teach and to encourage growth in 
positive attitudes. 


Background 


Lethbridge is a city of 60,000 in southwestern Alberta. Education has been shared 
for the past 100 years between the Public District and the Catholic District. The 
Lethbridge Catholic District has one high school, one junior high school, and six 
elementary schools with 170 teachers and 70 support staff to serve 2,700 students. 
The schools are places of hope where students are affirmed. The schools aim to 
provide excellent academic education in an environment of love and respect. 


Affective Domain 133 Lethbridge Catholic 


The importance of the affective domain to Lethbridge Catholic Schools is 
documented in its policies which aim: to create a positive climate; to help shape 
wholesome self-images in students; to build community through modeling; to set 
guidelines for spiritual growth and development; to acknowledge the dignity of 
helping children grow according to their abilities; to instill a responsible attitude 
toward the world and its people; to teach ways to relate acceptably to others; and to 
guide the process of evaluation, provide a sense of hope and success, recognize the 
need for continuous evaluation and trace the steps to take in order to celebrate 
learning. It is from this background that the Lethbridge Catholic School District 
came to the decision to search for signs of learning in the affective domain. 


Scope 


This project relates specifically to signs of learning in the affective domain. It 
defines the affective domain as that area of student development which includes 
attitudes, interests, appreciation, feelings, emotions and predispositions. This 
project makes no claim to finding signs of learning that relate to cognitive or skill 
development, however crossover may occur. 


Assumptions 


The project was based on the following assumptions and beliefs which both limited 
and directed the research. 


1. The affective domain includes attitudes, feelings, emotions and predispositions. 
Attitudes are of paramount importance to learning and they predispose us to 
behave in certain ways. Attitudes and values become evident in behavior. 


2. Intelligence is defined as behavior and is skill based. It is a dynamic 
characteristic that can be taught and learned. Intelligence is expressed in 
behavior. 


3. Tests and exams evaluate the acquisition of concepts and skills, but observation 
of behavior identifies the acquisition of attitudes. 


4. A student working to potential is dynamic in that the student keeps trying, 
continues to strive, and does not give up. Potential is not a preordained state to 
be filled, it is learning behavior which can be observed. 


5. Learning is celebrated when we tell what the student has learned, what is not yet 
learned and what plans we have to do more. In its ideal form, learning is a 
joyful process. 


6. The core of the curriculum is teaching behaviors which allow students to make 
informed choices. The act of choosing well requires a strong self-image. 
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Design 
Development 


The coordinator first reviewed the literature, then set out to discover what affective 
learning was seen as important to people in Lethbridge. Personal, open-ended, yet 
structured and focused, interviews were conducted with 100 people. Sampling was 
purposive rather than random. Those interviewed were students from grades 1 to 
12, parents, teachers, administrators, trustees, clergy, professionals, post- 
secondary educators, employers and people involved in correctional work. 
Responses from all of the constituents were recorded. 


The responses were sorted, categorized, classified, refined and grouped into five 
general areas. Affective learning was evident in: 


behaviors that showed self-worth, 

behaviors that dealt with interpersonal relationships, 

behaviors relating to world awareness and the expectations of the work 
world, 

behavior patterns of learning, 

behaviors relating to spiritual life, values and beliefs about purpose in life. 


Ae NP 


Field Testing 


One stage of field testing was the dissemination and explanation of the indicators to 
district teachers, parents, students, special interest groups, teachers outside the 
district, the school board and the media. 


The indicators and behaviors were translated into French for distribution to and 
response from the French immersion teachers in the district. 


The coordinator observed student behavior in district classrooms and reported on 
the many positive behaviors observed. Teachers reported that the observation of 
positive behavior had forced them to look at their class from a different vantage 
point, to look for the positive rather than dwell on the negative, and to be 
encouraged by the positive behaviors that were now so evident. 


As a result of the field test, it became obvious that teachers needed to focus attention 
on specific behaviors to avoid general overviews that lacked detail. Teachers needed 
to observe positive rather than negative behavior. They needed to be trained in 
differentiating between inference and observation. They needed to see the value of 
observing individual students not just the class as a whole. 


Field testing showed the following. 


1. Indicators of affective learning need not be totally comprehensive but they must 
be typical. 


2. Reporting of observation need not be of all affective indicators, for all students, 


at all times; but reporting of significant observations of individual students is 
helpful. 
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3. Reporting of affective learning is reporting of the observation of positive 
behavior; but observation of negative behavior is the basis for action plans to 
remedy the deficiencies. 


4. Checklists are useful reminders of behaviors observed, but they are too 
limiting. 


5. Published tests, instruments and measures of affective learning were not seen to 
be the most valid and pure measure of behavior. 


6. Behavior is observed in process and in product by teachers; behavior is 
observed by parents and peers; behavior is observed by students themselves 
when given time and freedom to reflect. 


7. In order to operationalize the indicators, there would need to be more specific 
descriptors of representative behaviors for each indicator. 


8. There would need to be ways to report observations in a positive manner. 
These ways would need to be easy, efficient and effective to use. 


Actions, typical of behaviors that teachers see, were collected from the field tests 
and from groups of teachers. These actions are not prescribed actions that students 
are required to display, but rather they are descriptions of typical actions of students 
who show affective learning. These actions were systematized and further refined 
by collaborating with practicing teachers. The descriptions expanded horizons and 
were included in a folder of possible actions to observe as evidence of the presence 
of a behavior. 


A Celebration of Learning 


Figure 1 presents a conceptual model of the relationship of all five areas of affective 
learning. The model represents the formation and growth of behaviors that display 
positive attitudes toward the SELF take place through the interrelated experiences of 
dealing with OTHERS, through a growing awareness of the WORLD, and through 
the process of LEARNING. The SPIRITUAL dimension unifies the other four, by 
identifying a purposiveness in life, its events and activities; it provides the hope that 
leads the learner on. 


Negative Schema 


We validated the trustworthiness of the list of positive behaviors by testing it 
against a mirror image of negative behaviors as is shown in Figures 2 and 3. 
Teachers were consulted to validate that the negative behaviors were those which 
show affective decay rather than affective growth. Negative behaviors were 
validated also from interview responses of behaviors which showed a lack of 
learning. We validated the behaviors in a practical sense in the classrooms of 
practicing teachers. We asked teachers to add to or delete behaviors which were not 
relevant to teaching and learning. The coordinator observed classrooms to validate 
the behaviors. The responses of teachers were used and the list of behaviors was 
refined, expanded, defined in more precise terms and validated again in the 
classroom. 
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Figure 1: The Conceptual Model 


Signs of Affective Learning 


The indicators of learning are Self Worth, Relating to Others, World Awareness, 
Learning, and Spiritual Life. The behaviors which are observed as evidence of 
these indicators follow. 


in SELF WORTH 

1. Demonstrates confidence, self-worth, and security by taking risks, accepting 
error with ease and good humor, maintaining a healthy life and taking pride in 
posture and appearance. 

2. Assumes responsibility by accepting the consequences of actions, accepting 
leadership, cooperating with responsible leaders, and demonstrating emotional 
well-being. 


3. Perseveres in developing gifts and talents through consistent effort, efficient use 
of time, and taking pride in accomplishments. 
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Figure 3: Indicators and Behaviors of Affective Decay 
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in RELATIONSHIP WITH OTHERS 


4. Shows consideration and respect for authority while understanding the 
legitimacy of dissent by choosing appropriate ways to agree or disagree, by 
responding to adult direction and by following rules. 


5. Appreciates the accomplishments of others by affirming and encouraging them, 
being sensitive to their feelings, and by accepting winning or losing in a 
gracious manner. 


6. Respects the rights, ideas and property of others by valuing diversity of ideas, 
caring for the property of others, accepting and understanding differences, and 
respecting the right of others to learn. 


in WORLD AWARENESS 


7. Displays a belief in the value of citizenship by contributing to the building of 
community, participating, sharing information or time or energy or resources, 
cooperating in groups, volunteering and becoming involved in community 
endeavors. 


8. Demonstrates a commitment to social justice and fair play by preserving the 
environment, conserving natural resources, using equipment and materials 
responsibly, and sharing generously. 


9. Exemplifies dependability and dedication by being present, punctual, prepared, 
attentive, diligent, good humored and able to work independently. 


in LEARNING 


10. Shows a desire to learn and to search for truth by being open and receptive to 


learning, affirming it and reflecting it with happiness, enthusiasm and satisfaction. 


11. Demonstrates self-direction and self-motivation by initiating activities, 
accepting challenges, extending knowledge, exploring possibilities, expanding 
horizons and creating. 


12. Displays intellectual curiosity by maintaining an open and inquisitive mind, 
being flexible to change, using skills of intelligence such as critical thinking 
skills, asking informed questions, and employing a variety of problem-solving 
techniques. 


in SPIRITUAL LIFE 
13. Demonstrates faith in a Supreme Being and in a system of values which 


distinguishes right from wrong by displaying truth, honesty, integrity, justice, 
and inner peace. 


14. Exemplifies hope, trust and purpose in life by being prudent and wise in 
decision making, and working toward a goal. 


15. Personifies love by showing acceptance, thankfulness, tolerance, compassion, 


reconciliation, and a sense of selflessness through considerate, responsible and 
trustworthy behavior. 
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Implementation 


The coordinator was actively involved in the implementation. She communicated 
the project and findings to district teachers formally on a school-by-school basis 
and informally on a one-to-one basis in staff rooms and classrooms. The 
coordinator was seen as a disinterested participant and teachers were free to discuss 
the negative, the positive, the interesting and the not so interesting results of 
introducing the concept of observing behavior as an authentic assessment tool. 


The demands of the classroom present some difficulty in gaining access to teachers’ 
time. The coordinator, a teacher, overcame this problem at one stage of the 
implementation by taking over the teaching and evaluating role of classrooms so 
that teachers could experience the benefits of observing the positive behaviors of 
their students in authentic situations. At another stage, she observed the behavior 
of students in classrooms and reported orally to the students and the teacher the type 
of learning behaviors that were observed. This exercise assisted teachers in looking 
for positive behaviors rather than seeing only negative behaviors which seem to 
jump out at you. 


The concept of celebrating learning by observing behavior must fit into an already 
existing system of assessment, evaluation and communication. To facilitate this, 
the coordinator became a participant in school based discussion, planning and 
redesigning of reporting systems. Since each of the eight schools in the district 
has its own reporting system, the coordinator incorporated the results of the 
project in a different manner for each group. 


Implementation required making distinctions between observing and inferring. 
The superintendent, who regularly communicates concepts, ideas, theories and 
assumptions to all teachers in the system through "Viewpoints" newsletters, 
exemplified observation and inference in one of the newsletters. 


During the implementation process it became evident that at the secondary level 
some teachers viewed affective learning as an add-on to the prescribed curriculum. 
They did not feel that there was time for anything extra in a semester. To answer 
this objection, the coordinator explained through personal contacts with both the 
junior high school and the senior high school that the affective domain is 
interwoven, interrelated and intertwined with every other domain to the extent 

that it is impossible to refrain from addressing affect. 


To assist the junior and senior high schools in implementing the reporting of 
positive behaviors apart from the cognitive and skill areas, the coordinator 
systematized the computer comments used on student progress reports. Comments 
were organized into administrative, achievement, skill and behavior sections. The 
behavior comments were divided into those which described positive actions 
observed and those which described negative actions observed. An attempt 

was made to delete inferences wherever possible. 


There was limited acceptance of the idea in the high school but much greater 
acceptance in the junior high school. However, some teachers in both junior and 
senior high schools adopted the concept of affirming positive behaviors with good 
news notes, rewriting the computer comments, and teaching behaviors in classes 
such as Career and Life Management, Religious Studies, and Health and Personal 
Development. We discontinued an attempt to describe growth in behavior that 
would differentiate between the nuances of behavior as the students mature. The 
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vast array of possibilities would be extremely cumbersome and would violate our 
criteria of practicality and simplicity. Since behavior does not develop in a linear 
fashion, we found it of no value to describe behavior as a series of benchmarks or 
standards representative of specific age groups. 


No portion of the project was mandated to teachers in Lethbridge Catholic Schools. 
It was offered to teachers to accept or reject in part or in whole. The only thing 
asked was that teachers continue to give feedback to the coordinator on how they 
used the behavior indicators, how they observed behavior, what instruments 
worked, what ideas did not work, what reactions they received from parents 

and students, and any suggestions they had to offer. 


The cardinal rule in implementing the project hinged on being able to show an 
immediate relationship between the theoretical and the practical aspects of it. The 
tension between the practical and the theoretical was lessened when it was made 
clear that the theory could be put into practice in a simple and efficient manner that 
would prove beneficial to teachers, students and parents. 


Instruments 


Research demonstrated that any procedure for observation of behaviors must meet 
several criteria. The procedure must be simple; the behaviors must be systematized 
and observable; the recording of observations must be time efficient; and the 
reporting of behaviors observed must be useful to teachers and helpful to students 
and parents. 


The instruments designed are simple; the procedure is simple and time efficient; 
the indicators are defensible. Observation can be used in the classroom, on the 
playground and at home. Behaviors are observable by parents, teachers, and 
students. Instruments for recording observations are open-ended. Instruments 
are designed to record and report positive behaviors in a constructive manner. 
Recording of observations is simple; reporting is simple yet affirming to students, 
informative for parents and useful for teachers. 


Each of the instruments developed is described below. 


1. A brochure to identify the 15 indicators of growth in behaviors. The behaviors 
are illustrated and displayed on five separate pages which identify them in chart 
form on graduated page lengths. The artist depicts birds displaying bird 
behaviors as they grow to maturity and become independent. 


2. A folder to identify seven representative actions for each of the 15 behaviors. 
The folder of 105 specific behaviors helps to focus observation. 


3. Acclass checklist of the 15 indicators to aliow for a quick method of recording 
reminders of observations. Later, the observations can be written in journals 
or anecdotes. 


4. Self-evaluations for students. There are separate versions for primary (grades 
1-3), elementary (grades 4-6) and secondary (grades 7-12) students. The 
primary student self-evaluations are five individual sheets covering each area of 
behavior. Each sheet asks eight questions to help students think about how they 
behave and to record their thoughts by coloring happy faces. The elementary 
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self-evaluation consists of one sheet that poses seven questions for each of the 
five areas. It also presents an opportunity to reflect on which behaviors they do 
well, which behaviors have improved and which ones still need improvement. 
The secondary self-evaluation is similar to the elementary one but the 
vocabulary and reflections take into account the maturity of the students. 


5. Good News Notes for students and parents. The notes are designed to allow 
teachers to give immediate feedback to students and parents when constructive 
and encouraging behavior has been observed at school. The carbon copy of the 
report allows teachers to keep a record of the observation for future assessment, 
discussion or conference. 


It may be of interest to note that the first attempt at preparing good news note 
pads met with less than encouraging results. The blank note pads were being 
used as just that - note pads. The redesigned note pads have a carbon copy, are 
addressed to student and parents, have part of the message printed on the note, 
and provide space for teachers to describe the behavior observed, the date and 
their signature. 


6. A presentation kit to enclose the instruments. The kit briefly describes how to 
use the kit, what assumptions are made, and the concept of the celebration of 
learning. 


Observations 


The purpose of the Lethbridge Catholic EQI Project was to find signs of learning in 
the affective domain. The project began by identifying indicators of the formation 
and presence of those attitudes deemed to be important goals of education. The 
project set out to design a system to monitor, record, and report behavior, the 
expression of attitudes. Unexpected purposes of the project became obvious; one 
was the need to train educators to observe positive behavior, another was the need 
to assist educators to teach and to model behaviors. 


We found specific observable behaviors indicating affective growth, that are valued 
by their stakeholders, promoted by Lethbridge Catholic School policies, mandated 
by Alberta Education, and supported in the literature. 


Some observations of the project follow. 


1. There is significant similarity between the responses of the various stakeholders 
to the interview questions. Stakeholders were asked to identify affective 
qualities which they valued as significant outcomes of schooling. The similarity 
of responses resulted in the indicators of affective learning identified by this 
project. 


2. The behaviors identified as indicators of affective learning are: 
acceptable - They have not been disputed. 

valid - They have shown trustworthiness. 

useful - They are practical in the classroom. 

transferable - Other school districts have adopted them. 
comprehensive - There have been no further suggestions. 
stimulating to teachers in their practice. 


eeeee 


Affective Domain 142 Lethbridge Catholic 


. In the implementation of the project, it is important to note that we did not 
mandate the use of the concept or the instruments. We offered them to teachers 
to use in the existing system to the degree that teachers found them useful. Our 
observation is that many teachers are ready to use or are using the behaviors and 
their representative actions to assess affective learning. They use the 
instruments: 

to teach behavior 

to describe behavior 

to talk to parents and assist them in parenting 

to use as a common reference in conferences 

to help write anecdotal reports 

to get ideas for report card comments 

to set objectives and goals for behavior 

to plan lessons and activities 

to focus their observations on specific actions 

to evaluate the affective areas of learning 

to describe growth and development of students in special education 
programs. 


. In the past we lacked an adequate vocabulary to simultaneously identify 
behavioral growth and deficiency. We recognize the need to identify the 
deficiency but not destroy the child. We have difficulty reporting that a student 
needs further development in behavior. The long list of behaviors prepares us to 
accept that behavior is not perfect in anyone, but that we can see growth in 
behavior if we choose to do so. 


. This project used observation of process and of product to assess behavior in an 
attempt to avoid the cognitive component of instrumental approaches and to 
provide a more pure measure of affective learning. The professional observation 
of teachers can be augmented by observations of parents, students and peers. 


Implications 


Behavior is dynamic and does not follow a linear pattern of development. 
Attempts to set standards by which behavior can be rated, graded or quantified 
would serve little purpose. 


The behavior of students is observed in places other than the school. It is 
important that multiple observers contribute to the assessment of affective 
learning. 


The procedure of using lists of descriptors to identify affective learning is 
instructive for students and parents. It is beneficial to teachers who can then 
base instructional and reporting decisions on direct observation. 


Including observation of behavior as authentic assessment of learning broadens 
the base of assessment and gives hope to students who have traditionally been 
assessed only on cognitive achievement end skill development. 


Good News Notes give immediate feedback to students on their behaviors 
which are conducive to learning. 
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6. The precision of the behaviors strengthens their use in observation and reduces 
the need for inference. 


7. There is resistance to implementing something new into an existing system. 
The new concept requires that old concepts change in shape or in significance. 
It requires flexibility in the system. In order for a new concept to be accepted 
and incorporated it must not only be simple, efficient, effective and beneficial, 
but it must be seen to be so. 


Recommendations 
For Lethbridge Catholic Schools 
1. That junior and senior high schools incorporate descriptors of the Signs of 
Affective Learning in their computerized comments for reporting student 
progress. 


2. That teachers acquaint themselves with the descriptors of affective learning 
presented in the Celebration of Learning kit. 


3. That teachers make use of the Good News Notes, provided in the kit, to affirm 
student behavior. 


4. That teachers make precise distinctions between observed student behavior and 
inferences based on observations. 


5. That teachers review the concept of A Celebration of Learning at student- 
parent-teacher interviews. 


6. That the district adopt the concept of A Celebration of Learning and the 
indicators of learning as the basis for describing student progress in the 
affective domain. 

For Other Districts 

7. That other districts evaluate the celebration of learning concept and supporting 
materials to determine their relevance to local needs. 

For Alberta Education 

8. That Alberta Education adopt the concept of A Celebration of Learning and the 
indicators of learning as the basis for describing student progress in the 


affective domain. 


9. That the department investigate the application of the concept of A Celebration 
of Learning to other domains. 
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10. That it adopt the definition of a student "working to potential" as a student 
who keeps trying and does not give up. Potential is a dynamic, continuing 
condition not a state to be realized. It follows that the criterion for checking 
any feedback on student performance is whether it allows and encourages 
students to keep trying. The skill of making choices becomes central for 
students. 


Follow-up 


A presentation kit including all relevant materials is being disseminated to the 
teachers of Lethbridge Catholic Schools. 


The research results are being presented to interests groups on request. 


We are presenting the results of the research project to the University of Lethbridge 
to be considered for possible use in their teacher training program. 


We are sending a copy of A Celebration of Learning to the Student Evaluation 
Branch of Alberta Education for their consideration. 


Conclusions 


We feel confident that the behaviors we identified as signs of learning in the 
affective domain make sense. They are research based and they are valued 
by the community. 


It is difficult to break the cultural bias that wants to compare. Some people still see 
rating, grading and comparing as necessary. Some people regard good grades as 
rewards and poor grades as goads to do better. Others believe there is a need for 
children to learn to fail and that mistakes are flaws which when repeated are the 
result of willfulness. Yet, we propose that the presence or absence of the identified 
positive behaviors is significant in itself. 


We find often that schools are applying evaluation concepts derived from assessing 
the cognitive and the psychomotor domains to assessment in the affective domain. 
This project has provided concepts and vocabulary specific to assessing affective 
learning. 


Negative behaviors are easily identified and are developed without assistance. 
Positive behaviors are not so easily identified because we assume them and they 

are subject to being overlooked. They are not always acknowledged or reported 

as signs of learning. There appeared to be no benefit in developing the negative side 
of the behavior chart any further than to use it to validate our findings. However, 
developing the positive side produced objectives for student outcomes and 
pedagogical consequences in teaching, demonstrating and modeling behaviors. 
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We experienced much interest in the project and anticipation of results for 
implementation. To fulfill the expectations of those who collaborated in the 
project, the results will be disseminated in a presentation kit which will include 
a brief description of the theory, a full description of the identified behaviors, 
master copies of the instruments developed, support for teaching and modeling 
behavior, and a user's guide. 


The findings of this project have a counter cultural quality in that they go against 
an educational concept derived perhaps from a confused religious concept. That 
confusion equated sin and error. Behaviors which were in error were sought out, 
exposed and expunged. But A Celebration of Learning would have the teacher 
look for opportunities to observe behavior, to find joy in the behavior of children 
in the act of learning, and to take pleasure in letting them know of it. 
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EQI Accomplishments and Next Steps 


This final chapter of Achieving Quality describes the major accomplishments of 
the EQI initiative, proposes potential provincial indicators, and outlines some 
next steps in expanding the adoption of educational quality indicators in school 
jurisdictions in Alberta. 


Accomplishments 


The Educational Quality Indicators initiative accomplished what it set out to do 
more than three years ago. Each of the past three years unfolded as originally 
conceived — an initial developmental year followed by field testing, and then a 
year to reflect upon the findings and refinement of the indicator systems. What 
has been learned can now be shared with others in Alberta and elsewhere. 


The major accomplishments of EQI include the development of enhanced 
indicator systems based on theory and practical experience; development of 
better ways to track educational improvement in Alberta; the collaboration 
between Alberta Education and the twelve school jurisdictions to develop 
indicator systems; and the involvement by the Conference of Alberta School 
Superintendents (CASS) in organizing the EQI conference. 


A Direction for Improvement 


The indicator systems developed by the participating school jurisdictions are 
based on theory and practice. An extensive review of the literature during the 
developmental phase ensured that teams were aware of conceptual frameworks 
and incorporated these into developing their indicator systems. The field tests 
tempered theory with practical realities to ensure that indicators that survived 
this phase were practical and had direct application for school districts. 


This direction for improvement incorporated a focus on student outcomes based 
on the goals of schooling, a balance among cognitive, affective and behavioral 
outcomes, multiple ways to collect and report information, and active involvement 
of educational partners (students, staff, parents, stakeholders, and the public). 


The local indicator systems have support within their communities because of 
the active involvement of partners in their development. Several models for 
assessing the effectiveness and efficiency of instructional programs, delivery 
systems, and overall system performance were developed. In addition, several 
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innovative alternative student assessment strategies were developed. These 
"products" can now be adapted by other districts and used to suit their local 


purposes. 


Products 


The project teams have developed many new approaches to assessing student 
learning and evaluating school systems, schools, programs and teachers. They are 
valuable additions to the existing set of measurement instruments because they are 
based on theory and refined through practice. Other districts can use these 
products to assess a broader range of outcomes. The major products developed 
are identified below. 


¢ The Lacombe/Rocky Mountain Project developed a comprehensive school 
system review process that can be adapted by other school jurisdictions. The 
data for student learning in intellectual, social, personal, and vocational areas 
can serve as points of reference for other districts wishing to measure these 
areas. The Handbook of Procedures for Implementing the School System 
Review Process is a complete guide for implementing the review. 


¢ The Grande Prairie Project developed prototype school and district profiles. 
The profiles report indicators of student achievement, school climate, quality 
of instruction, and funding. The district is working on a computer system to 
facilitate production of the profiles. 


¢ The Lethbridge Public Project developed a collaborative model for school and 
program evaluation that can be generalized to other programs and adapted 
by other districts. 


¢ The Spirit River Project identified 26 behaviors of effective teaching and 
grouped them in four areas — planning and preparation, instructional strategies, 
communication skills, and pupil/teacher relationships. Teachers were 
observed in 1991 and 1992 and composite performance profiles prepared 
for both years. The results serve to identify jurisdiction-level professional 
development needs. Other districts could adapt the procedures for local use. 


¢ The Fort McMurray Catholic Project incorporated the principles of outcome- 
based education (OBE) into grades 3 to 10 mathematics classes. OBE requires 
students to demonstrate mastery of the significant outcomes at each level. 
Alberta Education has released Every Child Can Learn which describes the 
philosophy and principles of OBE, and can serve as a collaborative guide for 
jurisdictions interested in this process. Fort McMurray Catholic and Alberta 
Education are planning a Canadian OBE conference to be held at the 
Fantasyland Hotel, May 17-19, 1993. 


¢ The Edmonton Public Project developed mechanisms for student portfolio 
assessment, the use of computer technology for communicating and storing 
information about student growth, and established and validated curriculum- 
based district level assessment strategies. 
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¢ The Calgary Public and Catholic Project developed indicators of quality 
student performance in art and mathematics. The EQ/-Art Handbook and 
Problem Solving Profiles provide exemplars of student performance in art 
and mathematics across elementary, junior and high school levels. 


¢ The Brooks Project explored ways to measure responsible student behavior. 
The project collected data from a variety of sources and involved the 
community in reaffirming its commitment to positive student behavior. 
Brooks’ Character in Our Community Beliefs incorporate these beliefs 
about responsible student behavior. 


¢ The Fort McMurray Public Project developed a three-level approach to 
measuring student social outcomes. The Social Skills Diagnostic Screen is a 
new instrument measuring social development in personal, interpersonal, 
social reasoning and classroom behavior. Products include an administration 
manual and a manual of activities to help students develop effective social 
skills. 


¢« The Lethbridge Catholic Project developed A Celebration of Learning which 
includes five indicators of affective behavior — self-worth, interpersonal 
relationships, world awareness, learning and spiritual life. Products include 
a presentation kit which contains a brochure, a class checklist, self-evaluation 
sheets for students, and Good News Notes. 


The results for selected indicators provide a point of reference for other districts 
that may wish to collect this type of information. For example, in addition to 
provincial achievement and diploma results, the School Subjects Attitude Scales 
were used by both the Lacombe/ Rocky Mountain and Fort McMurray Catholic 
projects to provide a measure of student attitudes. Participation rates in Alberta 
achievement tests and diploma exams are reported in the Lacombe/Rocky 
Mountain and Brooks summaries; these serve as a complementary indicator 

to achievement. 


Collaboration 


The EQI initiative has become a model for collaboration between government and 
school jurisdictions. Despite changes in project coordinators, superintendents, 
and three years of change in Alberta, the initiative met its target and was 
completed on time. 


The January 1993 EQI Conference marked the formal conclusion of the initiative, 
but the beginning of a new era of cooperation among educational partners 
interested in improving education. Organized by the Conference of Alberta 
School Superintendents, the conference provided an opportunity for the ten 
project teams to present the materials they developed and the results achieved 
over the past three years. More than 300 delegates from British Columbia 

to Newfoundland attended the conference. This was a major accomplishment 
because it was the first time that Alberta Education held a conference on a 
research endeavor, and the first time that CASS organized such an event for an 
Alberta Education initiative. The precedent bodes well for future cooperative 
projects between the department and CASS. 
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Communication 


EQI partners participated in symposia at two meetings of the Canadian Society 
for the Study of Education, one of the Learned Societies of Canada. In 1990, five 
project teams participated in the first EQI symposium in Victoria, BC at which 
they described their proposed indicator systems. In 1992, seven teams 
participated in the second symposium in Charlottetown, PEI where they shared 
preliminary results. Both were well attended and the feedback from the delegates 
was very positive. The proceedings of both symposia are available. The 
symposia helped participants clarify their projects and inform the Canadian 
academic community of the initiative. 


In addition to these two national symposia, teams participated in local workshops, 
professional development days, teachers’ conventions and conferences, and will 
continue to do so. Some also took part in national and international conferences. 
Requests for information about EQI have come from governments, school 
jurisdictions, universities and other organizations across Canada, the United 
States, Europe, Asia and Australia. 


Potential Provincial Indicators 


A major purpose of the EQI initiative was to develop a set of indicators for a 
provincial indicator system. At the fifth provincial meeting in January 1992, EQI 
partners discussed potential provincial indicators, effective practices, follow-up 
plans and issues. Participants recommended that provincial indicators should: 


¢ focus on student outcomes 

¢ be based on the goals of schooling and desirable personal characteristics 

* represent a balance among cognitive, affective and behavioral outcomes 

* provide corollary evidence (tests, surveys, interviews, records) 

¢ provide multiple perspectives (student, staff, parents, stakeholders, the public) 


Table 1 presents the indicators, measures and sources for the student outcomes for 
which there was consensus. Each indicator is tied to one or more of the goals of 
schooling. 


The value of the proposed indicators is twofold. Alberta's achievement testing 
and diploma examination programs already exist and provide districts with annual 
results for their students. Adding indicators of participation and completion 
provides a more comprehensive picture of student success. Whereas achievement 
provides a measure of academic excellence, participation provides an indication of 
equity. Together they provide a better picture of performance than either alone 
does. Moreover, evidence of exemplary products by students kept in a portfolio 
provides a measure of the diversity of student performance. The proposed 
indicators can, therefore, not only be aggregated and reported at the provincial 
level, but also at the district and school levels. Those involving student diversity 
and nonacademic areas can be collected and reported locally. Jurisdictions then 
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Goals! Indicator 


Students: 

1,2,3 achievement 
1,2 participation 
1,:2,.7 creativity 
1-7 graduation 
1,2 attitudes 

4 self-esteem 
1-7 satisfaction 
4-7 behavior 
5,6 responsibility 
4 fitness 


Table 1 


Potential Provincial Indicators 


Measures 


grades 3, 6, 9 achievement tests 
grade 12 diploma examinations 
Rutherford Scholarships 
performance assessment 


percentage of students taking 
courses and writing tests/exams 


evidence of exemplary work kept in 
portfolios (student products, videos) 


percentage of students who graduate 
from high schcol 


School Subjects Attitude Scales 2 
other attitude surveys 


available instruments 3 
student surveys 
observation 


satisfaction surveys 
personal interviews 


A Celebration of Learning 4 
Social Skills Diagnostic Screen > 


homework completion 
attendance 
vandalism 


Canada Fitness Award 


Source(s) 


students 


district 


students 


district 


students 


students 
teachers, parents 


students, teachers, public 


students, teachers, parents 


students 
district 
district 


students 


1 Goals of schooling: 1 communication, 2 content, 3 skills, 4 well-being, 5 citizenship, 6 world of work, 
7 desirable personal characteristics. Goals 1 and 3 are cognitive in nature, goal 2 contains both cognitive 
and affective aspects, and goals 4 to 7 essentially encompass all three domains. 


2 Students’ attitudes toward subjects are assessed on three factors: evaluation, usefulness, difficulty. 


3 There are several instruments available. Consult the EQI Inventory of Assessment Instruments. 


4 Student behaviors are described in five groups which show leaming in: self-worth, relating to others, 


world awareness, learning, spiritual life. See the Lethbridge Catholic report. 


5 Student behaviors are categorized into four areas: personal factors, interpersonal factors, social 


reasoning skills, classroom behaviors. See the Fort McMurray Public report. 
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are free to tailor their information reports to their publics based on local priorities 
and expectations. Therefore, accountability can be served both provincially and 
locally, while respecting local autonomy. 


Measuring indicators in multiple ways gives districts confidence in the 
information provided by a variety of sources. Provincial sources provide common 
indicators which ensure comparability among districts and provide an overall 
picture for the province. For example, when achievement is measured by 
achievement tests and diploma exams, Rutherford Scholarships, and performance 
assessments, districts can see how well students at different levels (grades 3, 6, 9, 
12) compare to the provincial average. Trends over time indicate whether 
improvement is occurring. 


Many of the products described in the previous section can be used by other 
districts immediately. For example, the Social Skills Diagnostic Screen can 
measure student social skills and A Celebration of Learning describes student 
affective behavior. Schools are microcosms of society, and social and affective 
skills are important areas for student growth and development. 


The project teams developed a common understanding of the implications of 
better information. This can now be shared with others to help them in 
planning, setting policies and practices, and making decisions. 


Next Steps 


The final project reports will be published as they are finalized. They will contain 
full documentation of the indicators, methods and findings of the individual 
projects. Alberta Education will release each report as it becomes available in 
the next few months. 


Alberta Education will also produce a synopsis of the EQI initiative for wide 
distribution to people interested in the handbooks and instruments that have been 
produced. These will be made available on a cost-recovery basis through the 
Learning Resources Distributing Centre. 


EQI teams are promoting their projects. They are presenting their materials to 
teachers in their districts, at professional development days, teachers' conventions 
and conferences. Project teams and Alberta Education staff will provide 
appropriate inservice activities to colleagues. Shared knowledge becomes 

a motivator to action. 


Alberta Education will provide leadership in promoting and extending the work of 
the EQI initiative throughout the province. Strategies to accomplish this include 
CASS zone meetings, teachers’ conventions, conferences, and workshops. 
Regional consortia will be promoted to integrate the products throughout the 
province. — 


EQI teams have also expressed interest in continuing to network and meet on a 


regular basis to maintain the momentum. Such meetings can become accessible 
to other districts wishing to participate. 
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Several issues remain to be resolved. For example, the EQI partners were unable 
to agree on a definition of a graduation rate since it can be calculated in many 
different ways. Other issues which remain include frequency of measurement, 
sampling of students, time, and data storage. These will be addressed in the 
coming months. 


Alberta Education will now work toward an integrated information system that 
includes a wider range of educational indicators to better reflect the goals of 
schooling. The interpretation and recommended directions of the local indicator 
projects, together with other provincial initiatives, provides a solid foundation for 
the implementation of an information system that measures the success of the 
educational enterprise. 


Concluding Statement 


The ownership and commitment that resulted from leaders developing local 
indicator systems, which reflect the values of their respective communities, 

have positive spin-off effects on the larger educational context in Alberta. The 
benefits of this three-year collaboration include the discussion among participants 
on the emerging set of indicators, the examination of issues, and the 
communication of results to diverse client groups. The information generated 
from the ten projects will assist Alberta Education to develop provincial 
indicators. The interpretation and recommended directions of the local indicator 
projects, together with other provincial initiatives, provides a solid foundation for 
the implementation of an information system that measures the success of the 
educational enterprise in the province. 


Educational indicators can provide better information to help improve education. 
They can provide a more balanced picture of the operation and outcomes of 
schooling, particularly if they focus on a broader range of desired outcomes. By 
focusing on student learning, and interpreting and reflecting on the findings, better 
decisions about schooling will result. 
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